2022-05-18T03:29:46.5835054Z Requested labels: linux.2xlarge
2022-05-18T03:29:46.5835146Z Job defined at: pytorch/pytorch/.github/workflows/_linux-test.yml@refs/heads/master
2022-05-18T03:29:46.5835169Z Waiting for a runner to pick up this job...
2022-05-18T03:29:48.3781320Z Job is about to start running on the runner: i-0244965c0907218b7 (repository)
2022-05-18T03:29:53.7903880Z Current runner version: '2.291.1'
2022-05-18T03:29:53.7909261Z Runner name: 'i-0244965c0907218b7'
2022-05-18T03:29:53.7909827Z Runner group name: 'Default'
2022-05-18T03:29:53.7910328Z Machine name: 'ip-10-0-5-106'
2022-05-18T03:29:53.7912288Z ##[group]GITHUB_TOKEN Permissions
2022-05-18T03:29:53.7913015Z Actions: write
2022-05-18T03:29:53.7913339Z Checks: write
2022-05-18T03:29:53.7913579Z Contents: write
2022-05-18T03:29:53.7913911Z Deployments: write
2022-05-18T03:29:53.7914205Z Discussions: write
2022-05-18T03:29:53.7914447Z Issues: write
2022-05-18T03:29:53.7914742Z Metadata: read
2022-05-18T03:29:53.7915030Z Packages: write
2022-05-18T03:29:53.7915275Z Pages: write
2022-05-18T03:29:53.7915568Z PullRequests: write
2022-05-18T03:29:53.7915913Z RepositoryProjects: write
2022-05-18T03:29:53.7916195Z SecurityEvents: write
2022-05-18T03:29:53.7916530Z Statuses: write
2022-05-18T03:29:53.7916810Z ##[endgroup]
2022-05-18T03:29:53.7919750Z Secret source: Actions
2022-05-18T03:29:53.7920389Z Prepare workflow directory
2022-05-18T03:29:54.0067410Z Prepare all required actions
2022-05-18T03:29:54.0233520Z Getting action download info
2022-05-18T03:29:54.1930858Z Download action repository 'pytorch/pytorch@master' (SHA:acf7136a525422459d97d5f993e30afdff18b1b9)
2022-05-18T03:29:56.5879558Z Download action repository 'nick-fields/retry@71062288b76e2b6214ebde0e673ce0de1755740a' (SHA:71062288b76e2b6214ebde0e673ce0de1755740a)
2022-05-18T03:29:56.6772975Z Download action repository 'seemethere/upload-artifact-s3@v4' (SHA:c1c31f57581a11fe6d4d052da6276adb2df71f1e)
2022-05-18T03:29:56.8850991Z Getting action download info
2022-05-18T03:29:57.1047163Z Download action repository 'malfet/checkout@silent-checkout' (SHA:f63e9e15406be6060f159846cd2e098f759c5246)
2022-05-18T03:29:57.2710228Z Getting action download info
2022-05-18T03:29:57.4799092Z ##[group]Run pytorch/pytorch/.github/actions/checkout-pytorch@master
2022-05-18T03:29:57.4799379Z with:
2022-05-18T03:29:57.4799571Z   submodules: recursive
2022-05-18T03:29:57.4799754Z   fetch-depth: 0
2022-05-18T03:29:57.4799935Z env:
2022-05-18T03:29:57.4800097Z   IN_CI: 1
2022-05-18T03:29:57.4800246Z   IS_GHA: 1
2022-05-18T03:29:57.4800445Z   GIT_DEFAULT_BRANCH: master
2022-05-18T03:29:57.4800639Z ##[endgroup]
2022-05-18T03:29:57.5010328Z ##[group]Run echo "${GITHUB_WORKSPACE}"
2022-05-18T03:29:57.5010601Z echo "${GITHUB_WORKSPACE}"
2022-05-18T03:29:57.5010825Z if [ -z "${NO_SUDO}" ]; then
2022-05-18T03:29:57.5011045Z   sudo rm -rf "${GITHUB_WORKSPACE}"
2022-05-18T03:29:57.5011227Z else
2022-05-18T03:29:57.5011418Z   rm -rf "${GITHUB_WORKSPACE}"
2022-05-18T03:29:57.5011606Z fi
2022-05-18T03:29:57.5011776Z mkdir "${GITHUB_WORKSPACE}"
2022-05-18T03:29:57.5026985Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-05-18T03:29:57.5027216Z env:
2022-05-18T03:29:57.5027377Z   IN_CI: 1
2022-05-18T03:29:57.5027539Z   IS_GHA: 1
2022-05-18T03:29:57.5027708Z   GIT_DEFAULT_BRANCH: master
2022-05-18T03:29:57.5027894Z   NO_SUDO: 
2022-05-18T03:29:57.5028074Z ##[endgroup]
2022-05-18T03:29:57.5186465Z /home/ec2-user/actions-runner/_work/pytorch/pytorch
2022-05-18T03:30:00.0049912Z ##[group]Run malfet/checkout@silent-checkout
2022-05-18T03:30:00.0050184Z with:
2022-05-18T03:30:00.0050416Z   ref: 3b2375291aab7b48442f2e6fb1ef66cebc761e24
2022-05-18T03:30:00.0050656Z   fetch-depth: 0
2022-05-18T03:30:00.0050860Z   submodules: recursive
2022-05-18T03:30:00.0051079Z   quiet-checkout: true
2022-05-18T03:30:00.0051312Z   repository: pytorch/pytorch
2022-05-18T03:30:00.0051727Z   token: ***
2022-05-18T03:30:00.0051918Z   ssh-strict: true
2022-05-18T03:30:00.0052142Z   persist-credentials: true
2022-05-18T03:30:00.0052366Z   clean: true
2022-05-18T03:30:00.0052546Z   lfs: false
2022-05-18T03:30:00.0052768Z   set-safe-directory: true
2022-05-18T03:30:00.0052974Z env:
2022-05-18T03:30:00.0053145Z   IN_CI: 1
2022-05-18T03:30:00.0053334Z   IS_GHA: 1
2022-05-18T03:30:00.0053546Z   GIT_DEFAULT_BRANCH: master
2022-05-18T03:30:00.0053749Z ##[endgroup]
2022-05-18T03:30:00.1165933Z Syncing repository: pytorch/pytorch
2022-05-18T03:30:00.1167751Z ##[group]Getting Git version info
2022-05-18T03:30:00.1168461Z Working directory is '/home/ec2-user/actions-runner/_work/pytorch/pytorch'
2022-05-18T03:30:00.1168966Z [command]/usr/bin/git version
2022-05-18T03:30:00.1169180Z git version 2.32.0
2022-05-18T03:30:00.1169846Z ##[endgroup]
2022-05-18T03:30:00.1182539Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/cea6103b-531c-4490-87bd-f94372948894' before making global git config changes
2022-05-18T03:30:00.1183291Z Adding repository directory to the temporary git global config as a safe directory
2022-05-18T03:30:00.1188510Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch
2022-05-18T03:30:00.1226679Z Deleting the contents of '/home/ec2-user/actions-runner/_work/pytorch/pytorch'
2022-05-18T03:30:00.1230120Z ##[group]Initializing the repository
2022-05-18T03:30:00.1234512Z [command]/usr/bin/git init /home/ec2-user/actions-runner/_work/pytorch/pytorch
2022-05-18T03:30:00.1343575Z hint: Using 'master' as the name for the initial branch. This default branch name
2022-05-18T03:30:00.1344169Z hint: is subject to change. To configure the initial branch name to use in all
2022-05-18T03:30:00.1344670Z hint: of your new repositories, which will suppress this warning, call:
2022-05-18T03:30:00.1345076Z hint: 
2022-05-18T03:30:00.1345656Z hint: 	git config --global init.defaultBranch <name>
2022-05-18T03:30:00.1346048Z hint: 
2022-05-18T03:30:00.1346641Z hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and
2022-05-18T03:30:00.1347180Z hint: 'development'. The just-created branch can be renamed via this command:
2022-05-18T03:30:00.1347556Z hint: 
2022-05-18T03:30:00.1348149Z hint: 	git branch -m <name>
2022-05-18T03:30:00.1348845Z Initialized empty Git repository in /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/
2022-05-18T03:30:00.1356660Z [command]/usr/bin/git remote add origin https://github.com/pytorch/pytorch
2022-05-18T03:30:00.1384869Z ##[endgroup]
2022-05-18T03:30:00.1385255Z ##[group]Disabling automatic garbage collection
2022-05-18T03:30:00.1389547Z [command]/usr/bin/git config --local gc.auto 0
2022-05-18T03:30:00.1414235Z ##[endgroup]
2022-05-18T03:30:00.1414557Z ##[group]Setting up auth
2022-05-18T03:30:00.1421567Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand
2022-05-18T03:30:00.1449667Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :
2022-05-18T03:30:00.1691487Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader
2022-05-18T03:30:00.1721556Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :
2022-05-18T03:30:00.1970103Z [command]/usr/bin/git config --local http.https://github.com/.extraheader AUTHORIZATION: basic ***
2022-05-18T03:30:00.2010049Z ##[endgroup]
2022-05-18T03:30:00.2010430Z ##[group]Fetching the repository
2022-05-18T03:30:00.2016028Z [command]/usr/bin/git -c protocol.version=2 fetch --prune --quiet --no-recurse-submodules origin +refs/heads/*:refs/remotes/origin/* +refs/tags/*:refs/tags/*
2022-05-18T03:30:42.4629798Z [command]/usr/bin/git rev-parse --verify --quiet 3b2375291aab7b48442f2e6fb1ef66cebc761e24^{object}
2022-05-18T03:30:42.4655340Z 3b2375291aab7b48442f2e6fb1ef66cebc761e24
2022-05-18T03:30:42.4660574Z ##[endgroup]
2022-05-18T03:30:42.4661170Z ##[group]Determining the checkout info
2022-05-18T03:30:42.4661918Z ##[endgroup]
2022-05-18T03:30:42.4662509Z ##[group]Checking out the ref
2022-05-18T03:30:42.4667311Z [command]/usr/bin/git checkout --quiet --force 3b2375291aab7b48442f2e6fb1ef66cebc761e24
2022-05-18T03:30:43.6267916Z ##[endgroup]
2022-05-18T03:30:43.6268520Z ##[group]Setting up auth for fetching submodules
2022-05-18T03:30:43.6274074Z [command]/usr/bin/git config --global http.https://github.com/.extraheader AUTHORIZATION: basic ***
2022-05-18T03:30:43.6318761Z [command]/usr/bin/git config --global --unset-all url.https://github.com/.insteadOf
2022-05-18T03:30:43.6346748Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf git@github.com:
2022-05-18T03:30:43.6373125Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf org-21003710@github.com:
2022-05-18T03:30:43.6397908Z ##[endgroup]
2022-05-18T03:30:43.6398260Z ##[group]Fetching submodules
2022-05-18T03:30:43.6402843Z [command]/usr/bin/git submodule sync --recursive
2022-05-18T03:30:43.6665104Z [command]/usr/bin/git -c protocol.version=2 submodule update --init --force --recursive
2022-05-18T03:30:43.6917841Z Submodule 'android/libs/fbjni' (https://github.com/facebookincubator/fbjni.git) registered for path 'android/libs/fbjni'
2022-05-18T03:30:43.6918817Z Submodule 'third_party/NNPACK_deps/FP16' (https://github.com/Maratyszcza/FP16.git) registered for path 'third_party/FP16'
2022-05-18T03:30:43.6920575Z Submodule 'third_party/NNPACK_deps/FXdiv' (https://github.com/Maratyszcza/FXdiv.git) registered for path 'third_party/FXdiv'
2022-05-18T03:30:43.6922748Z Submodule 'third_party/NNPACK' (https://github.com/Maratyszcza/NNPACK.git) registered for path 'third_party/NNPACK'
2022-05-18T03:30:43.6925028Z Submodule 'third_party/QNNPACK' (https://github.com/pytorch/QNNPACK) registered for path 'third_party/QNNPACK'
2022-05-18T03:30:43.6927397Z Submodule 'third_party/XNNPACK' (https://github.com/google/XNNPACK.git) registered for path 'third_party/XNNPACK'
2022-05-18T03:30:43.6930080Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/benchmark'
2022-05-18T03:30:43.6932552Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo.git) registered for path 'third_party/cpuinfo'
2022-05-18T03:30:43.6935274Z Submodule 'third_party/cub' (https://github.com/NVlabs/cub.git) registered for path 'third_party/cub'
2022-05-18T03:30:43.6938111Z Submodule 'third_party/cudnn_frontend' (https://github.com/NVIDIA/cudnn-frontend.git) registered for path 'third_party/cudnn_frontend'
2022-05-18T03:30:43.6940917Z Submodule 'third_party/eigen' (https://gitlab.com/libeigen/eigen.git) registered for path 'third_party/eigen'
2022-05-18T03:30:43.6944140Z Submodule 'third_party/fbgemm' (https://github.com/pytorch/fbgemm) registered for path 'third_party/fbgemm'
2022-05-18T03:30:43.6947296Z Submodule 'third_party/flatbuffers' (https://github.com/google/flatbuffers.git) registered for path 'third_party/flatbuffers'
2022-05-18T03:30:43.6950371Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/fmt'
2022-05-18T03:30:43.6953799Z Submodule 'third_party/foxi' (https://github.com/houseroad/foxi.git) registered for path 'third_party/foxi'
2022-05-18T03:30:43.6957182Z Submodule 'third_party/gemmlowp/gemmlowp' (https://github.com/google/gemmlowp.git) registered for path 'third_party/gemmlowp/gemmlowp'
2022-05-18T03:30:43.6960644Z Submodule 'third_party/gloo' (https://github.com/facebookincubator/gloo) registered for path 'third_party/gloo'
2022-05-18T03:30:43.6964242Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/googletest'
2022-05-18T03:30:43.6967859Z Submodule 'third_party/ideep' (https://github.com/intel/ideep) registered for path 'third_party/ideep'
2022-05-18T03:30:43.6971901Z Submodule 'third_party/ios-cmake' (https://github.com/Yangqing/ios-cmake.git) registered for path 'third_party/ios-cmake'
2022-05-18T03:30:43.6975611Z Submodule 'third_party/kineto' (https://github.com/pytorch/kineto) registered for path 'third_party/kineto'
2022-05-18T03:30:43.6979581Z Submodule 'third_party/nccl/nccl' (https://github.com/NVIDIA/nccl) registered for path 'third_party/nccl/nccl'
2022-05-18T03:30:43.6983877Z Submodule 'third_party/neon2sse' (https://github.com/intel/ARM_NEON_2_x86_SSE.git) registered for path 'third_party/neon2sse'
2022-05-18T03:30:43.6988019Z Submodule 'third_party/onnx' (https://github.com/onnx/onnx.git) registered for path 'third_party/onnx'
2022-05-18T03:30:43.6992370Z Submodule 'third_party/onnx-tensorrt' (https://github.com/onnx/onnx-tensorrt) registered for path 'third_party/onnx-tensorrt'
2022-05-18T03:30:43.6996746Z Submodule 'third_party/pocketfft' (https://github.com/mreineck/pocketfft) registered for path 'third_party/pocketfft'
2022-05-18T03:30:43.7001257Z Submodule 'third_party/protobuf' (https://github.com/protocolbuffers/protobuf.git) registered for path 'third_party/protobuf'
2022-05-18T03:30:43.7005805Z Submodule 'third_party/NNPACK_deps/psimd' (https://github.com/Maratyszcza/psimd.git) registered for path 'third_party/psimd'
2022-05-18T03:30:43.7010474Z Submodule 'third_party/NNPACK_deps/pthreadpool' (https://github.com/Maratyszcza/pthreadpool.git) registered for path 'third_party/pthreadpool'
2022-05-18T03:30:43.7015198Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/pybind11'
2022-05-18T03:30:43.7020076Z Submodule 'third_party/python-enum' (https://github.com/PeachPy/enum34.git) registered for path 'third_party/python-enum'
2022-05-18T03:30:43.7025233Z Submodule 'third_party/python-peachpy' (https://github.com/Maratyszcza/PeachPy.git) registered for path 'third_party/python-peachpy'
2022-05-18T03:30:43.7030193Z Submodule 'third_party/python-six' (https://github.com/benjaminp/six.git) registered for path 'third_party/python-six'
2022-05-18T03:30:43.7035416Z Submodule 'third_party/sleef' (https://github.com/shibatch/sleef) registered for path 'third_party/sleef'
2022-05-18T03:30:43.7040758Z Submodule 'third_party/tbb' (https://github.com/01org/tbb) registered for path 'third_party/tbb'
2022-05-18T03:30:43.7046071Z Submodule 'third_party/tensorpipe' (https://github.com/pytorch/tensorpipe.git) registered for path 'third_party/tensorpipe'
2022-05-18T03:30:43.7051676Z Submodule 'third_party/zstd' (https://github.com/facebook/zstd.git) registered for path 'third_party/zstd'
2022-05-18T03:30:43.7104838Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/android/libs/fbjni'...
2022-05-18T03:30:43.9384157Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FP16'...
2022-05-18T03:30:44.1177113Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FXdiv'...
2022-05-18T03:30:44.2899697Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/NNPACK'...
2022-05-18T03:30:44.5116915Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/QNNPACK'...
2022-05-18T03:30:44.7412758Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/XNNPACK'...
2022-05-18T03:30:49.3375432Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/benchmark'...
2022-05-18T03:30:49.6627350Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpuinfo'...
2022-05-18T03:30:50.0760465Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cub'...
2022-05-18T03:30:51.2711503Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cudnn_frontend'...
2022-05-18T03:30:52.1803615Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/eigen'...
2022-05-18T03:30:56.2935374Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm'...
2022-05-18T03:30:56.7916772Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/flatbuffers'...
2022-05-18T03:30:57.7158654Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fmt'...
2022-05-18T03:30:58.6690720Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/foxi'...
2022-05-18T03:30:58.8386812Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gemmlowp/gemmlowp'...
2022-05-18T03:30:59.2139320Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gloo'...
2022-05-18T03:30:59.4600340Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/googletest'...
2022-05-18T03:31:00.3251265Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep'...
2022-05-18T03:31:00.6507964Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ios-cmake'...
2022-05-18T03:31:00.8202349Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto'...
2022-05-18T03:31:02.0560634Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nccl/nccl'...
2022-05-18T03:31:02.3898408Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/neon2sse'...
2022-05-18T03:31:02.7478122Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx'...
2022-05-18T03:31:03.9265730Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt'...
2022-05-18T03:31:04.2731691Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pocketfft'...
2022-05-18T03:31:04.4638637Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf'...
2022-05-18T03:31:08.5836836Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/psimd'...
2022-05-18T03:31:08.7510526Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pthreadpool'...
2022-05-18T03:31:08.9343267Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pybind11'...
2022-05-18T03:31:09.5669127Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/python-enum'...
2022-05-18T03:31:09.7634819Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/python-peachpy'...
2022-05-18T03:31:10.0051909Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/python-six'...
2022-05-18T03:31:10.2499600Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/sleef'...
2022-05-18T03:31:10.7245233Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tbb'...
2022-05-18T03:31:12.1948149Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe'...
2022-05-18T03:31:12.5907346Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/zstd'...
2022-05-18T03:31:14.3630930Z Submodule path 'android/libs/fbjni': checked out '7e1e1fe3858c63c251c637ae41a20de425dde96f'
2022-05-18T03:31:14.3939172Z Submodule path 'third_party/FP16': checked out '4dfe081cf6bcd15db339cf2680b9281b8451eeb3'
2022-05-18T03:31:14.4218584Z Submodule path 'third_party/FXdiv': checked out 'b408327ac2a15ec3e43352421954f5b1967701d1'
2022-05-18T03:31:14.4625434Z Submodule path 'third_party/NNPACK': checked out 'c07e3a0400713d546e0dea2d5466dd22ea389c73'
2022-05-18T03:31:14.5030088Z Submodule path 'third_party/QNNPACK': checked out '7d2a4e9931a82adc3814275b6219a03e24e36b4c'
2022-05-18T03:31:15.0790779Z Submodule path 'third_party/XNNPACK': checked out 'ae108ef49aa5623b896fc93d4298c49d1750d9ba'
2022-05-18T03:31:15.1197218Z Submodule path 'third_party/benchmark': checked out 'e991355c02b93fe17713efe04cbc2e278e00fdbd'
2022-05-18T03:31:15.2312609Z Submodule path 'third_party/cpuinfo': checked out '5916273f79a21551890fd3d56fc5375a78d1598d'
2022-05-18T03:31:15.2842511Z Submodule path 'third_party/cub': checked out 'd106ddb991a56c3df1b6d51b2409e36ba8181ce4'
2022-05-18T03:31:15.5658780Z Submodule path 'third_party/cudnn_frontend': checked out '43709ab96c47e26eebcdac72f93f946d44ceffa8'
2022-05-18T03:31:15.8055282Z Submodule path 'third_party/eigen': checked out '3147391d946bb4b6c68edd901f2add6ac1f31f8c'
2022-05-18T03:31:15.8664835Z Submodule path 'third_party/fbgemm': checked out '2e9be65810107a9595da717f95d21924b73be833'
2022-05-18T03:31:15.8705356Z Submodule 'third_party/asmjit' (https://github.com/asmjit/asmjit.git) registered for path 'third_party/fbgemm/third_party/asmjit'
2022-05-18T03:31:15.8706858Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo) registered for path 'third_party/fbgemm/third_party/cpuinfo'
2022-05-18T03:31:15.8709444Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/fbgemm/third_party/googletest'
2022-05-18T03:31:15.8746635Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/asmjit'...
2022-05-18T03:31:16.5269966Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cpuinfo'...
2022-05-18T03:31:16.9599174Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/googletest'...
2022-05-18T03:31:17.8830408Z Submodule path 'third_party/fbgemm/third_party/asmjit': checked out '8b35b4cffb62ecb58a903bf91cb7537d7a672211'
2022-05-18T03:31:17.9953041Z Submodule path 'third_party/fbgemm/third_party/cpuinfo': checked out 'ed8b86a253800bafdb7b25c5c399f91bff9cb1f3'
2022-05-18T03:31:18.0669718Z Submodule path 'third_party/fbgemm/third_party/googletest': checked out 'cbf019de22c8dd37b2108da35b2748fd702d1796'
2022-05-18T03:31:18.1660504Z Submodule path 'third_party/flatbuffers': checked out 'd0cede9c90c5257537c293517a21376408b549fa'
2022-05-18T03:31:18.2194362Z Submodule path 'third_party/fmt': checked out 'cd4af11efc9c622896a3e4cb599fa28668ca3d05'
2022-05-18T03:31:18.2497380Z Submodule path 'third_party/foxi': checked out 'c278588e34e535f0bb8f00df3880d26928038cad'
2022-05-18T03:31:18.3078011Z Submodule path 'third_party/gemmlowp/gemmlowp': checked out '3fb5c176c17c765a3492cd2f0321b0dab712f350'
2022-05-18T03:31:18.3503668Z Submodule path 'third_party/gloo': checked out 'c22a5cfba94edf8ea4f53a174d38aa0c629d070f'
2022-05-18T03:31:18.4133951Z Submodule path 'third_party/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929'
2022-05-18T03:31:18.4443419Z Submodule path 'third_party/ideep': checked out '02b17c5748c9349dcc586c359af800c684d9b1ab'
2022-05-18T03:31:18.4485856Z Submodule 'mkl-dnn' (https://github.com/intel/mkl-dnn.git) registered for path 'third_party/ideep/mkl-dnn'
2022-05-18T03:31:18.4520854Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep/mkl-dnn'...
2022-05-18T03:31:23.4718236Z Submodule path 'third_party/ideep/mkl-dnn': checked out '888a87a954e4fddb4d81fd10858eb834f2441b46'
2022-05-18T03:31:23.4769938Z Submodule 'third_party/oneDNN' (https://github.com/oneapi-src/oneDNN.git) registered for path 'third_party/ideep/mkl-dnn/third_party/oneDNN'
2022-05-18T03:31:23.4808263Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep/mkl-dnn/third_party/oneDNN'...
2022-05-18T03:31:28.4963855Z Submodule path 'third_party/ideep/mkl-dnn/third_party/oneDNN': checked out '52b5f107dd9cf10910aaa19cb47f3abf9b349815'
2022-05-18T03:31:28.5281163Z Submodule path 'third_party/ios-cmake': checked out '8abaed637d56f1337d6e1d2c4026e25c1eade724'
2022-05-18T03:31:28.6334672Z Submodule path 'third_party/kineto': checked out 'b2b48c00c6e5bd8e807e2231adb229db6a1d1c22'
2022-05-18T03:31:28.6377787Z Submodule 'libkineto/third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/fmt'
2022-05-18T03:31:28.6378820Z Submodule 'libkineto/third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/googletest'
2022-05-18T03:31:28.6414710Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/fmt'...
2022-05-18T03:31:29.6291327Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/googletest'...
2022-05-18T03:31:30.5405818Z Submodule path 'third_party/kineto/libkineto/third_party/fmt': checked out '2591ab91c3898c9f6544fff04660276537d32ffd'
2022-05-18T03:31:30.6084776Z Submodule path 'third_party/kineto/libkineto/third_party/googletest': checked out '7aca84427f224eeed3144123d5230d5871e93347'
2022-05-18T03:31:30.6472719Z Submodule path 'third_party/nccl/nccl': checked out '7e515921295adaab72adf56ea71a0fafb0ecb5f3'
2022-05-18T03:31:30.6802791Z Submodule path 'third_party/neon2sse': checked out '97a126f08ce318023be604d03f88bf0820a9464a'
2022-05-18T03:31:30.9115834Z Submodule path 'third_party/onnx': checked out '96046b8ccfb8e6fa82f6b2b34b3d56add2e8849c'
2022-05-18T03:31:30.9168221Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/onnx/third_party/benchmark'
2022-05-18T03:31:30.9169304Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/onnx/third_party/pybind11'
2022-05-18T03:31:30.9215433Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/benchmark'...
2022-05-18T03:31:31.2462719Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/pybind11'...
2022-05-18T03:31:31.9223656Z Submodule path 'third_party/onnx/third_party/benchmark': checked out 'e776aa0275e293707b6a0901e0e8d8a8a3679508'
2022-05-18T03:31:31.9711596Z Submodule path 'third_party/onnx/third_party/pybind11': checked out '59a2ac2745d8a57ac94c6accced73620d59fb844'
2022-05-18T03:31:32.0055375Z Submodule path 'third_party/onnx-tensorrt': checked out 'c153211418a7c57ce071d9ce2a41f8d1c85a878f'
2022-05-18T03:31:32.0094490Z Submodule 'third_party/onnx' (https://github.com/onnx/onnx.git) registered for path 'third_party/onnx-tensorrt/third_party/onnx'
2022-05-18T03:31:32.0128278Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx'...
2022-05-18T03:31:33.3605916Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx': checked out '765f5ee823a67a866f4bd28a9860e81f3c811ce8'
2022-05-18T03:31:33.3659928Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2022-05-18T03:31:33.3660767Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2022-05-18T03:31:33.3702320Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'...
2022-05-18T03:31:33.6996401Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'...
2022-05-18T03:31:34.3670056Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark': checked out 'e776aa0275e293707b6a0901e0e8d8a8a3679508'
2022-05-18T03:31:34.4471000Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11': checked out 'a1041190c8b8ff0cd9e2f0752248ad5e3789ea0c'
2022-05-18T03:31:34.4520972Z Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2022-05-18T03:31:34.4558016Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'...
2022-05-18T03:31:34.6508107Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5'
2022-05-18T03:31:34.6817845Z Submodule path 'third_party/pocketfft': checked out 'ea778e37710c07723435b1be58235996d1d43a5a'
2022-05-18T03:31:34.9321974Z Submodule path 'third_party/protobuf': checked out 'd1eca4e4b421cd2997495c4b4e65cea6be4e9b8a'
2022-05-18T03:31:34.9376324Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/protobuf/third_party/benchmark'
2022-05-18T03:31:34.9377356Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/protobuf/third_party/googletest'
2022-05-18T03:31:34.9410280Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/benchmark'...
2022-05-18T03:31:35.2708709Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/googletest'...
2022-05-18T03:31:36.1512390Z Submodule path 'third_party/protobuf/third_party/benchmark': checked out '5b7683f49e1e9223cf9927b24f6fd3d6bd82e3f8'
2022-05-18T03:31:36.2327502Z Submodule path 'third_party/protobuf/third_party/googletest': checked out '5ec7f0c4a113e2f18ac2c6cc7df51ad6afc24081'
2022-05-18T03:31:36.2617170Z Submodule path 'third_party/psimd': checked out '072586a71b55b7f8c584153d223e95687148a900'
2022-05-18T03:31:36.2917526Z Submodule path 'third_party/pthreadpool': checked out 'a134dd5d4cee80cce15db81a72e7f929d71dd413'
2022-05-18T03:31:36.3390257Z Submodule path 'third_party/pybind11': checked out '8de7772cc72daca8e947b79b83fea46214931604'
2022-05-18T03:31:36.3669675Z Submodule path 'third_party/python-enum': checked out '4cfedc426c4e2fc52e3f5c2b4297e15ed8d6b8c7'
2022-05-18T03:31:36.4133458Z Submodule path 'third_party/python-peachpy': checked out '07d8fde8ac45d7705129475c0f94ed8925b93473'
2022-05-18T03:31:36.4424595Z Submodule path 'third_party/python-six': checked out '15e31431af97e5e64b80af0a3f598d382bcdd49a'
2022-05-18T03:31:36.5032415Z Submodule path 'third_party/sleef': checked out 'e0a003ee838b75d11763aa9c3ef17bf71a725bff'
2022-05-18T03:31:36.6216388Z Submodule path 'third_party/tbb': checked out 'a51a90bc609bb73db8ea13841b5cf7aa4344d4a9'
2022-05-18T03:31:36.6656424Z Submodule path 'third_party/tensorpipe': checked out '52791a2fd214b2a9dc5759d36725909c1daa7f2e'
2022-05-18T03:31:36.6696747Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/tensorpipe/third_party/googletest'
2022-05-18T03:31:36.6698416Z Submodule 'third_party/libnop' (https://github.com/google/libnop.git) registered for path 'third_party/tensorpipe/third_party/libnop'
2022-05-18T03:31:36.6700716Z Submodule 'third_party/libuv' (https://github.com/libuv/libuv.git) registered for path 'third_party/tensorpipe/third_party/libuv'
2022-05-18T03:31:36.6703372Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/tensorpipe/third_party/pybind11'
2022-05-18T03:31:36.6739347Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/googletest'...
2022-05-18T03:31:37.5247657Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libnop'...
2022-05-18T03:31:37.7351678Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libuv'...
2022-05-18T03:31:38.7007898Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11'...
2022-05-18T03:31:39.4081199Z Submodule path 'third_party/tensorpipe/third_party/googletest': checked out 'aee0f9d9b5b87796ee8a0ab26b7587ec30e8858e'
2022-05-18T03:31:39.4417679Z Submodule path 'third_party/tensorpipe/third_party/libnop': checked out '910b55815be16109f04f4180e9adee14fb4ce281'
2022-05-18T03:31:39.5213188Z Submodule path 'third_party/tensorpipe/third_party/libuv': checked out '1dff88e5161cba5c59276d2070d2e304e4dcb242'
2022-05-18T03:31:39.5665126Z Submodule path 'third_party/tensorpipe/third_party/pybind11': checked out 'a23996fce38ff6ccfbcdc09f1e63f2c4be5ea2ef'
2022-05-18T03:31:39.5710880Z Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2022-05-18T03:31:39.5745310Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11/tools/clang'...
2022-05-18T03:31:39.7812674Z Submodule path 'third_party/tensorpipe/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5'
2022-05-18T03:31:39.9220387Z Submodule path 'third_party/zstd': checked out 'aec56a52fbab207fc639a1937d1e708a282edca8'
2022-05-18T03:31:39.9296005Z [command]/usr/bin/git submodule foreach --recursive git config --local gc.auto 0
2022-05-18T03:31:39.9550790Z Entering 'android/libs/fbjni'
2022-05-18T03:31:39.9584698Z Entering 'third_party/FP16'
2022-05-18T03:31:39.9616973Z Entering 'third_party/FXdiv'
2022-05-18T03:31:39.9650789Z Entering 'third_party/NNPACK'
2022-05-18T03:31:39.9684481Z Entering 'third_party/QNNPACK'
2022-05-18T03:31:39.9719324Z Entering 'third_party/XNNPACK'
2022-05-18T03:31:39.9762937Z Entering 'third_party/benchmark'
2022-05-18T03:31:39.9795971Z Entering 'third_party/cpuinfo'
2022-05-18T03:31:39.9830471Z Entering 'third_party/cub'
2022-05-18T03:31:39.9863617Z Entering 'third_party/cudnn_frontend'
2022-05-18T03:31:39.9901840Z Entering 'third_party/eigen'
2022-05-18T03:31:39.9936854Z Entering 'third_party/fbgemm'
2022-05-18T03:31:39.9970542Z Entering 'third_party/fbgemm/third_party/asmjit'
2022-05-18T03:31:40.0003624Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2022-05-18T03:31:40.0038632Z Entering 'third_party/fbgemm/third_party/googletest'
2022-05-18T03:31:40.0073596Z Entering 'third_party/flatbuffers'
2022-05-18T03:31:40.0109447Z Entering 'third_party/fmt'
2022-05-18T03:31:40.0143433Z Entering 'third_party/foxi'
2022-05-18T03:31:40.0176739Z Entering 'third_party/gemmlowp/gemmlowp'
2022-05-18T03:31:40.0210283Z Entering 'third_party/gloo'
2022-05-18T03:31:40.0244548Z Entering 'third_party/googletest'
2022-05-18T03:31:40.0279580Z Entering 'third_party/ideep'
2022-05-18T03:31:40.0312514Z Entering 'third_party/ideep/mkl-dnn'
2022-05-18T03:31:40.0346904Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN'
2022-05-18T03:31:40.0386250Z Entering 'third_party/ios-cmake'
2022-05-18T03:31:40.0419912Z Entering 'third_party/kineto'
2022-05-18T03:31:40.0454113Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2022-05-18T03:31:40.0487785Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2022-05-18T03:31:40.0521928Z Entering 'third_party/nccl/nccl'
2022-05-18T03:31:40.0555705Z Entering 'third_party/neon2sse'
2022-05-18T03:31:40.0589442Z Entering 'third_party/onnx'
2022-05-18T03:31:40.0634054Z Entering 'third_party/onnx/third_party/benchmark'
2022-05-18T03:31:40.0667637Z Entering 'third_party/onnx/third_party/pybind11'
2022-05-18T03:31:40.0702680Z Entering 'third_party/onnx-tensorrt'
2022-05-18T03:31:40.0735714Z Entering 'third_party/onnx-tensorrt/third_party/onnx'
2022-05-18T03:31:40.0774568Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2022-05-18T03:31:40.0807986Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2022-05-18T03:31:40.0842505Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2022-05-18T03:31:40.0879582Z Entering 'third_party/pocketfft'
2022-05-18T03:31:40.0913037Z Entering 'third_party/protobuf'
2022-05-18T03:31:40.0950361Z Entering 'third_party/protobuf/third_party/benchmark'
2022-05-18T03:31:40.0983607Z Entering 'third_party/protobuf/third_party/googletest'
2022-05-18T03:31:40.1018295Z Entering 'third_party/psimd'
2022-05-18T03:31:40.1051749Z Entering 'third_party/pthreadpool'
2022-05-18T03:31:40.1084648Z Entering 'third_party/pybind11'
2022-05-18T03:31:40.1118791Z Entering 'third_party/python-enum'
2022-05-18T03:31:40.1151956Z Entering 'third_party/python-peachpy'
2022-05-18T03:31:40.1184950Z Entering 'third_party/python-six'
2022-05-18T03:31:40.1218071Z Entering 'third_party/sleef'
2022-05-18T03:31:40.1251357Z Entering 'third_party/tbb'
2022-05-18T03:31:40.1285830Z Entering 'third_party/tensorpipe'
2022-05-18T03:31:40.1321280Z Entering 'third_party/tensorpipe/third_party/googletest'
2022-05-18T03:31:40.1354034Z Entering 'third_party/tensorpipe/third_party/libnop'
2022-05-18T03:31:40.1386221Z Entering 'third_party/tensorpipe/third_party/libuv'
2022-05-18T03:31:40.1419290Z Entering 'third_party/tensorpipe/third_party/pybind11'
2022-05-18T03:31:40.1452914Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2022-05-18T03:31:40.1487600Z Entering 'third_party/zstd'
2022-05-18T03:31:40.1530058Z ##[endgroup]
2022-05-18T03:31:40.1530653Z ##[group]Persisting credentials for submodules
2022-05-18T03:31:40.1538245Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'url\.https\:\/\/github\.com\/\.insteadOf' && git config --local --unset-all 'url.https://github.com/.insteadOf' || :
2022-05-18T03:31:40.1802119Z Entering 'android/libs/fbjni'
2022-05-18T03:31:40.1834550Z Entering 'third_party/FP16'
2022-05-18T03:31:40.1868456Z Entering 'third_party/FXdiv'
2022-05-18T03:31:40.1901076Z Entering 'third_party/NNPACK'
2022-05-18T03:31:40.1933483Z Entering 'third_party/QNNPACK'
2022-05-18T03:31:40.1966567Z Entering 'third_party/XNNPACK'
2022-05-18T03:31:40.2009769Z Entering 'third_party/benchmark'
2022-05-18T03:31:40.2042879Z Entering 'third_party/cpuinfo'
2022-05-18T03:31:40.2075171Z Entering 'third_party/cub'
2022-05-18T03:31:40.2108338Z Entering 'third_party/cudnn_frontend'
2022-05-18T03:31:40.2145560Z Entering 'third_party/eigen'
2022-05-18T03:31:40.2179737Z Entering 'third_party/fbgemm'
2022-05-18T03:31:40.2211423Z Entering 'third_party/fbgemm/third_party/asmjit'
2022-05-18T03:31:40.2243833Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2022-05-18T03:31:40.2275919Z Entering 'third_party/fbgemm/third_party/googletest'
2022-05-18T03:31:40.2309919Z Entering 'third_party/flatbuffers'
2022-05-18T03:31:40.2344168Z Entering 'third_party/fmt'
2022-05-18T03:31:40.2376483Z Entering 'third_party/foxi'
2022-05-18T03:31:40.2409421Z Entering 'third_party/gemmlowp/gemmlowp'
2022-05-18T03:31:40.2442516Z Entering 'third_party/gloo'
2022-05-18T03:31:40.2475373Z Entering 'third_party/googletest'
2022-05-18T03:31:40.2508346Z Entering 'third_party/ideep'
2022-05-18T03:31:40.2541042Z Entering 'third_party/ideep/mkl-dnn'
2022-05-18T03:31:40.2574872Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN'
2022-05-18T03:31:40.2613517Z Entering 'third_party/ios-cmake'
2022-05-18T03:31:40.2646678Z Entering 'third_party/kineto'
2022-05-18T03:31:40.2682715Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2022-05-18T03:31:40.2715690Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2022-05-18T03:31:40.2749226Z Entering 'third_party/nccl/nccl'
2022-05-18T03:31:40.2781670Z Entering 'third_party/neon2sse'
2022-05-18T03:31:40.2814975Z Entering 'third_party/onnx'
2022-05-18T03:31:40.2858343Z Entering 'third_party/onnx/third_party/benchmark'
2022-05-18T03:31:40.2890810Z Entering 'third_party/onnx/third_party/pybind11'
2022-05-18T03:31:40.2925283Z Entering 'third_party/onnx-tensorrt'
2022-05-18T03:31:40.2959578Z Entering 'third_party/onnx-tensorrt/third_party/onnx'
2022-05-18T03:31:40.2996861Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2022-05-18T03:31:40.3029086Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2022-05-18T03:31:40.3061520Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2022-05-18T03:31:40.3097506Z Entering 'third_party/pocketfft'
2022-05-18T03:31:40.3129975Z Entering 'third_party/protobuf'
2022-05-18T03:31:40.3166497Z Entering 'third_party/protobuf/third_party/benchmark'
2022-05-18T03:31:40.3198056Z Entering 'third_party/protobuf/third_party/googletest'
2022-05-18T03:31:40.3231172Z Entering 'third_party/psimd'
2022-05-18T03:31:40.3263174Z Entering 'third_party/pthreadpool'
2022-05-18T03:31:40.3294978Z Entering 'third_party/pybind11'
2022-05-18T03:31:40.3327560Z Entering 'third_party/python-enum'
2022-05-18T03:31:40.3359393Z Entering 'third_party/python-peachpy'
2022-05-18T03:31:40.3392004Z Entering 'third_party/python-six'
2022-05-18T03:31:40.3424003Z Entering 'third_party/sleef'
2022-05-18T03:31:40.3456620Z Entering 'third_party/tbb'
2022-05-18T03:31:40.3490755Z Entering 'third_party/tensorpipe'
2022-05-18T03:31:40.3523657Z Entering 'third_party/tensorpipe/third_party/googletest'
2022-05-18T03:31:40.3554500Z Entering 'third_party/tensorpipe/third_party/libnop'
2022-05-18T03:31:40.3587653Z Entering 'third_party/tensorpipe/third_party/libuv'
2022-05-18T03:31:40.3619779Z Entering 'third_party/tensorpipe/third_party/pybind11'
2022-05-18T03:31:40.3651179Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2022-05-18T03:31:40.3685865Z Entering 'third_party/zstd'
2022-05-18T03:31:40.3731200Z [command]/usr/bin/git submodule foreach --recursive git config --local 'http.https://github.com/.extraheader' 'AUTHORIZATION: basic ***' && git config --local --show-origin --name-only --get-regexp remote.origin.url
2022-05-18T03:31:40.3984950Z Entering 'android/libs/fbjni'
2022-05-18T03:31:40.4014515Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/android/libs/fbjni/config	remote.origin.url
2022-05-18T03:31:40.4028346Z Entering 'third_party/FP16'
2022-05-18T03:31:40.4058993Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FP16/config	remote.origin.url
2022-05-18T03:31:40.4072607Z Entering 'third_party/FXdiv'
2022-05-18T03:31:40.4102659Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FXdiv/config	remote.origin.url
2022-05-18T03:31:40.4116104Z Entering 'third_party/NNPACK'
2022-05-18T03:31:40.4146948Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK/config	remote.origin.url
2022-05-18T03:31:40.4161336Z Entering 'third_party/QNNPACK'
2022-05-18T03:31:40.4191403Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/QNNPACK/config	remote.origin.url
2022-05-18T03:31:40.4204953Z Entering 'third_party/XNNPACK'
2022-05-18T03:31:40.4237054Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/XNNPACK/config	remote.origin.url
2022-05-18T03:31:40.4260128Z Entering 'third_party/benchmark'
2022-05-18T03:31:40.4292640Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/benchmark/config	remote.origin.url
2022-05-18T03:31:40.4306225Z Entering 'third_party/cpuinfo'
2022-05-18T03:31:40.4336901Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpuinfo/config	remote.origin.url
2022-05-18T03:31:40.4350798Z Entering 'third_party/cub'
2022-05-18T03:31:40.4381213Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cub/config	remote.origin.url
2022-05-18T03:31:40.4395492Z Entering 'third_party/cudnn_frontend'
2022-05-18T03:31:40.4427085Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cudnn_frontend/config	remote.origin.url
2022-05-18T03:31:40.4446449Z Entering 'third_party/eigen'
2022-05-18T03:31:40.4477394Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/eigen/config	remote.origin.url
2022-05-18T03:31:40.4493233Z Entering 'third_party/fbgemm'
2022-05-18T03:31:40.4524483Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/config	remote.origin.url
2022-05-18T03:31:40.4538501Z Entering 'third_party/fbgemm/third_party/asmjit'
2022-05-18T03:31:40.4568734Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/asmjit/config	remote.origin.url
2022-05-18T03:31:40.4581976Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2022-05-18T03:31:40.4612340Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cpuinfo/config	remote.origin.url
2022-05-18T03:31:40.4626108Z Entering 'third_party/fbgemm/third_party/googletest'
2022-05-18T03:31:40.4656745Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/googletest/config	remote.origin.url
2022-05-18T03:31:40.4673030Z Entering 'third_party/flatbuffers'
2022-05-18T03:31:40.4703738Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flatbuffers/config	remote.origin.url
2022-05-18T03:31:40.4719059Z Entering 'third_party/fmt'
2022-05-18T03:31:40.4749263Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fmt/config	remote.origin.url
2022-05-18T03:31:40.4762609Z Entering 'third_party/foxi'
2022-05-18T03:31:40.4792897Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/foxi/config	remote.origin.url
2022-05-18T03:31:40.4805987Z Entering 'third_party/gemmlowp/gemmlowp'
2022-05-18T03:31:40.4837908Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gemmlowp/gemmlowp/config	remote.origin.url
2022-05-18T03:31:40.4851703Z Entering 'third_party/gloo'
2022-05-18T03:31:40.4882085Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gloo/config	remote.origin.url
2022-05-18T03:31:40.4895483Z Entering 'third_party/googletest'
2022-05-18T03:31:40.4926327Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/googletest/config	remote.origin.url
2022-05-18T03:31:40.4939536Z Entering 'third_party/ideep'
2022-05-18T03:31:40.4971001Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/config	remote.origin.url
2022-05-18T03:31:40.4984462Z Entering 'third_party/ideep/mkl-dnn'
2022-05-18T03:31:40.5015026Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/config	remote.origin.url
2022-05-18T03:31:40.5030123Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN'
2022-05-18T03:31:40.5060648Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/modules/third_party/oneDNN/config	remote.origin.url
2022-05-18T03:31:40.5080892Z Entering 'third_party/ios-cmake'
2022-05-18T03:31:40.5112757Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ios-cmake/config	remote.origin.url
2022-05-18T03:31:40.5126246Z Entering 'third_party/kineto'
2022-05-18T03:31:40.5157377Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/config	remote.origin.url
2022-05-18T03:31:40.5171418Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2022-05-18T03:31:40.5202782Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/fmt/config	remote.origin.url
2022-05-18T03:31:40.5216784Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2022-05-18T03:31:40.5247817Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/googletest/config	remote.origin.url
2022-05-18T03:31:40.5262670Z Entering 'third_party/nccl/nccl'
2022-05-18T03:31:40.5294039Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nccl/nccl/config	remote.origin.url
2022-05-18T03:31:40.5307872Z Entering 'third_party/neon2sse'
2022-05-18T03:31:40.5338567Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/neon2sse/config	remote.origin.url
2022-05-18T03:31:40.5352417Z Entering 'third_party/onnx'
2022-05-18T03:31:40.5383285Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/config	remote.origin.url
2022-05-18T03:31:40.5406046Z Entering 'third_party/onnx/third_party/benchmark'
2022-05-18T03:31:40.5437168Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/benchmark/config	remote.origin.url
2022-05-18T03:31:40.5450810Z Entering 'third_party/onnx/third_party/pybind11'
2022-05-18T03:31:40.5482153Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/pybind11/config	remote.origin.url
2022-05-18T03:31:40.5497501Z Entering 'third_party/onnx-tensorrt'
2022-05-18T03:31:40.5529108Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/config	remote.origin.url
2022-05-18T03:31:40.5542318Z Entering 'third_party/onnx-tensorrt/third_party/onnx'
2022-05-18T03:31:40.5572699Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/config	remote.origin.url
2022-05-18T03:31:40.5590512Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2022-05-18T03:31:40.5621155Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/modules/third_party/benchmark/config	remote.origin.url
2022-05-18T03:31:40.5635236Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2022-05-18T03:31:40.5665630Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/modules/third_party/pybind11/config	remote.origin.url
2022-05-18T03:31:40.5679125Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2022-05-18T03:31:40.5710930Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/modules/third_party/pybind11/modules/tools/clang/config	remote.origin.url
2022-05-18T03:31:40.5728806Z Entering 'third_party/pocketfft'
2022-05-18T03:31:40.5759220Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pocketfft/config	remote.origin.url
2022-05-18T03:31:40.5792532Z Entering 'third_party/protobuf'
2022-05-18T03:31:40.5804732Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/config	remote.origin.url
2022-05-18T03:31:40.5821648Z Entering 'third_party/protobuf/third_party/benchmark'
2022-05-18T03:31:40.5852636Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/benchmark/config	remote.origin.url
2022-05-18T03:31:40.5866389Z Entering 'third_party/protobuf/third_party/googletest'
2022-05-18T03:31:40.5896723Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/googletest/config	remote.origin.url
2022-05-18T03:31:40.5911694Z Entering 'third_party/psimd'
2022-05-18T03:31:40.5942171Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/psimd/config	remote.origin.url
2022-05-18T03:31:40.5955594Z Entering 'third_party/pthreadpool'
2022-05-18T03:31:40.5987211Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/pthreadpool/config	remote.origin.url
2022-05-18T03:31:40.6000901Z Entering 'third_party/pybind11'
2022-05-18T03:31:40.6032281Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pybind11/config	remote.origin.url
2022-05-18T03:31:40.6046153Z Entering 'third_party/python-enum'
2022-05-18T03:31:40.6076902Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-enum/config	remote.origin.url
2022-05-18T03:31:40.6090559Z Entering 'third_party/python-peachpy'
2022-05-18T03:31:40.6121666Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-peachpy/config	remote.origin.url
2022-05-18T03:31:40.6134875Z Entering 'third_party/python-six'
2022-05-18T03:31:40.6164699Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-six/config	remote.origin.url
2022-05-18T03:31:40.6178054Z Entering 'third_party/sleef'
2022-05-18T03:31:40.6208287Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/sleef/config	remote.origin.url
2022-05-18T03:31:40.6222019Z Entering 'third_party/tbb'
2022-05-18T03:31:40.6252651Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tbb/config	remote.origin.url
2022-05-18T03:31:40.6268094Z Entering 'third_party/tensorpipe'
2022-05-18T03:31:40.6298346Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/config	remote.origin.url
2022-05-18T03:31:40.6311991Z Entering 'third_party/tensorpipe/third_party/googletest'
2022-05-18T03:31:40.6341529Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/googletest/config	remote.origin.url
2022-05-18T03:31:40.6354918Z Entering 'third_party/tensorpipe/third_party/libnop'
2022-05-18T03:31:40.6385590Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libnop/config	remote.origin.url
2022-05-18T03:31:40.6398641Z Entering 'third_party/tensorpipe/third_party/libuv'
2022-05-18T03:31:40.6428605Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libuv/config	remote.origin.url
2022-05-18T03:31:40.6442192Z Entering 'third_party/tensorpipe/third_party/pybind11'
2022-05-18T03:31:40.6472066Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/config	remote.origin.url
2022-05-18T03:31:40.6484891Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2022-05-18T03:31:40.6515720Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/modules/tools/clang/config	remote.origin.url
2022-05-18T03:31:40.6532066Z Entering 'third_party/zstd'
2022-05-18T03:31:40.6562329Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/zstd/config	remote.origin.url
2022-05-18T03:31:40.7093718Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'git@github.com:'
2022-05-18T03:31:40.7345757Z Entering 'android/libs/fbjni'
2022-05-18T03:31:40.7378328Z Entering 'third_party/FP16'
2022-05-18T03:31:40.7412647Z Entering 'third_party/FXdiv'
2022-05-18T03:31:40.7446772Z Entering 'third_party/NNPACK'
2022-05-18T03:31:40.7481612Z Entering 'third_party/QNNPACK'
2022-05-18T03:31:40.7514943Z Entering 'third_party/XNNPACK'
2022-05-18T03:31:40.7558000Z Entering 'third_party/benchmark'
2022-05-18T03:31:40.7591888Z Entering 'third_party/cpuinfo'
2022-05-18T03:31:40.7625545Z Entering 'third_party/cub'
2022-05-18T03:31:40.7658763Z Entering 'third_party/cudnn_frontend'
2022-05-18T03:31:40.7696540Z Entering 'third_party/eigen'
2022-05-18T03:31:40.7731660Z Entering 'third_party/fbgemm'
2022-05-18T03:31:40.7765098Z Entering 'third_party/fbgemm/third_party/asmjit'
2022-05-18T03:31:40.7797956Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2022-05-18T03:31:40.7831222Z Entering 'third_party/fbgemm/third_party/googletest'
2022-05-18T03:31:40.7865432Z Entering 'third_party/flatbuffers'
2022-05-18T03:31:40.7900737Z Entering 'third_party/fmt'
2022-05-18T03:31:40.7933864Z Entering 'third_party/foxi'
2022-05-18T03:31:40.7966825Z Entering 'third_party/gemmlowp/gemmlowp'
2022-05-18T03:31:40.8000658Z Entering 'third_party/gloo'
2022-05-18T03:31:40.8033949Z Entering 'third_party/googletest'
2022-05-18T03:31:40.8066830Z Entering 'third_party/ideep'
2022-05-18T03:31:40.8099052Z Entering 'third_party/ideep/mkl-dnn'
2022-05-18T03:31:40.8133596Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN'
2022-05-18T03:31:40.8172204Z Entering 'third_party/ios-cmake'
2022-05-18T03:31:40.8205526Z Entering 'third_party/kineto'
2022-05-18T03:31:40.8239461Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2022-05-18T03:31:40.8274061Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2022-05-18T03:31:40.8309102Z Entering 'third_party/nccl/nccl'
2022-05-18T03:31:40.8342333Z Entering 'third_party/neon2sse'
2022-05-18T03:31:40.8374904Z Entering 'third_party/onnx'
2022-05-18T03:31:40.8417831Z Entering 'third_party/onnx/third_party/benchmark'
2022-05-18T03:31:40.8450997Z Entering 'third_party/onnx/third_party/pybind11'
2022-05-18T03:31:40.8486378Z Entering 'third_party/onnx-tensorrt'
2022-05-18T03:31:40.8519094Z Entering 'third_party/onnx-tensorrt/third_party/onnx'
2022-05-18T03:31:40.8557049Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2022-05-18T03:31:40.8590143Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2022-05-18T03:31:40.8623436Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2022-05-18T03:31:40.8660129Z Entering 'third_party/pocketfft'
2022-05-18T03:31:40.8693018Z Entering 'third_party/protobuf'
2022-05-18T03:31:40.8729995Z Entering 'third_party/protobuf/third_party/benchmark'
2022-05-18T03:31:40.8763759Z Entering 'third_party/protobuf/third_party/googletest'
2022-05-18T03:31:40.8798491Z Entering 'third_party/psimd'
2022-05-18T03:31:40.8832471Z Entering 'third_party/pthreadpool'
2022-05-18T03:31:40.8865925Z Entering 'third_party/pybind11'
2022-05-18T03:31:40.8899406Z Entering 'third_party/python-enum'
2022-05-18T03:31:40.8933263Z Entering 'third_party/python-peachpy'
2022-05-18T03:31:40.8966326Z Entering 'third_party/python-six'
2022-05-18T03:31:40.8999358Z Entering 'third_party/sleef'
2022-05-18T03:31:40.9033952Z Entering 'third_party/tbb'
2022-05-18T03:31:40.9068744Z Entering 'third_party/tensorpipe'
2022-05-18T03:31:40.9102298Z Entering 'third_party/tensorpipe/third_party/googletest'
2022-05-18T03:31:40.9134997Z Entering 'third_party/tensorpipe/third_party/libnop'
2022-05-18T03:31:40.9168642Z Entering 'third_party/tensorpipe/third_party/libuv'
2022-05-18T03:31:40.9204365Z Entering 'third_party/tensorpipe/third_party/pybind11'
2022-05-18T03:31:40.9236807Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2022-05-18T03:31:40.9271702Z Entering 'third_party/zstd'
2022-05-18T03:31:40.9316326Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'org-21003710@github.com:'
2022-05-18T03:31:40.9569812Z Entering 'android/libs/fbjni'
2022-05-18T03:31:40.9602927Z Entering 'third_party/FP16'
2022-05-18T03:31:40.9636857Z Entering 'third_party/FXdiv'
2022-05-18T03:31:40.9670425Z Entering 'third_party/NNPACK'
2022-05-18T03:31:40.9703496Z Entering 'third_party/QNNPACK'
2022-05-18T03:31:40.9737209Z Entering 'third_party/XNNPACK'
2022-05-18T03:31:40.9780468Z Entering 'third_party/benchmark'
2022-05-18T03:31:40.9813650Z Entering 'third_party/cpuinfo'
2022-05-18T03:31:40.9847779Z Entering 'third_party/cub'
2022-05-18T03:31:40.9880665Z Entering 'third_party/cudnn_frontend'
2022-05-18T03:31:40.9920342Z Entering 'third_party/eigen'
2022-05-18T03:31:40.9956289Z Entering 'third_party/fbgemm'
2022-05-18T03:31:40.9988269Z Entering 'third_party/fbgemm/third_party/asmjit'
2022-05-18T03:31:41.0021190Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2022-05-18T03:31:41.0054286Z Entering 'third_party/fbgemm/third_party/googletest'
2022-05-18T03:31:41.0087690Z Entering 'third_party/flatbuffers'
2022-05-18T03:31:41.0122665Z Entering 'third_party/fmt'
2022-05-18T03:31:41.0154810Z Entering 'third_party/foxi'
2022-05-18T03:31:41.0187099Z Entering 'third_party/gemmlowp/gemmlowp'
2022-05-18T03:31:41.0219709Z Entering 'third_party/gloo'
2022-05-18T03:31:41.0252949Z Entering 'third_party/googletest'
2022-05-18T03:31:41.0286220Z Entering 'third_party/ideep'
2022-05-18T03:31:41.0324148Z Entering 'third_party/ideep/mkl-dnn'
2022-05-18T03:31:41.0357727Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN'
2022-05-18T03:31:41.0396294Z Entering 'third_party/ios-cmake'
2022-05-18T03:31:41.0429940Z Entering 'third_party/kineto'
2022-05-18T03:31:41.0463283Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2022-05-18T03:31:41.0495874Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2022-05-18T03:31:41.0530349Z Entering 'third_party/nccl/nccl'
2022-05-18T03:31:41.0564194Z Entering 'third_party/neon2sse'
2022-05-18T03:31:41.0597667Z Entering 'third_party/onnx'
2022-05-18T03:31:41.0642033Z Entering 'third_party/onnx/third_party/benchmark'
2022-05-18T03:31:41.0675182Z Entering 'third_party/onnx/third_party/pybind11'
2022-05-18T03:31:41.0710165Z Entering 'third_party/onnx-tensorrt'
2022-05-18T03:31:41.0742496Z Entering 'third_party/onnx-tensorrt/third_party/onnx'
2022-05-18T03:31:41.0779411Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2022-05-18T03:31:41.0812969Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2022-05-18T03:31:41.0846310Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2022-05-18T03:31:41.0883421Z Entering 'third_party/pocketfft'
2022-05-18T03:31:41.0917000Z Entering 'third_party/protobuf'
2022-05-18T03:31:41.0953890Z Entering 'third_party/protobuf/third_party/benchmark'
2022-05-18T03:31:41.0987389Z Entering 'third_party/protobuf/third_party/googletest'
2022-05-18T03:31:41.1022558Z Entering 'third_party/psimd'
2022-05-18T03:31:41.1056628Z Entering 'third_party/pthreadpool'
2022-05-18T03:31:41.1091245Z Entering 'third_party/pybind11'
2022-05-18T03:31:41.1125605Z Entering 'third_party/python-enum'
2022-05-18T03:31:41.1159382Z Entering 'third_party/python-peachpy'
2022-05-18T03:31:41.1192501Z Entering 'third_party/python-six'
2022-05-18T03:31:41.1229238Z Entering 'third_party/sleef'
2022-05-18T03:31:41.1262664Z Entering 'third_party/tbb'
2022-05-18T03:31:41.1297994Z Entering 'third_party/tensorpipe'
2022-05-18T03:31:41.1331926Z Entering 'third_party/tensorpipe/third_party/googletest'
2022-05-18T03:31:41.1364834Z Entering 'third_party/tensorpipe/third_party/libnop'
2022-05-18T03:31:41.1398775Z Entering 'third_party/tensorpipe/third_party/libuv'
2022-05-18T03:31:41.1432287Z Entering 'third_party/tensorpipe/third_party/pybind11'
2022-05-18T03:31:41.1465171Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2022-05-18T03:31:41.1500112Z Entering 'third_party/zstd'
2022-05-18T03:31:41.1541465Z ##[endgroup]
2022-05-18T03:31:41.1579246Z [command]/usr/bin/git log -1 --format='%H'
2022-05-18T03:31:41.1604268Z '3b2375291aab7b48442f2e6fb1ef66cebc761e24'
2022-05-18T03:31:41.1720119Z Prepare all required actions
2022-05-18T03:31:41.1743138Z ##[group]Run ./.github/actions/setup-linux
2022-05-18T03:31:41.1743419Z env:
2022-05-18T03:31:41.1743617Z   IN_CI: 1
2022-05-18T03:31:41.1743851Z   IS_GHA: 1
2022-05-18T03:31:41.1744114Z   GIT_DEFAULT_BRANCH: master
2022-05-18T03:31:41.1744291Z ##[endgroup]
2022-05-18T03:31:41.1757018Z ##[group]Run set -euo pipefail
2022-05-18T03:31:41.1757254Z set -euo pipefail
2022-05-18T03:31:41.1757453Z function get_ec2_metadata() {
2022-05-18T03:31:41.1757701Z   # Pulled from instance metadata endpoint for EC2
2022-05-18T03:31:41.1758055Z   # see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html
2022-05-18T03:31:41.1758350Z   category=$1
2022-05-18T03:31:41.1758595Z   curl -fsSL "http://169.254.169.254/latest/meta-data/${category}"
2022-05-18T03:31:41.1758817Z }
2022-05-18T03:31:41.1759034Z echo "ami-id: $(get_ec2_metadata ami-id)"
2022-05-18T03:31:41.1759290Z echo "instance-id: $(get_ec2_metadata instance-id)"
2022-05-18T03:31:41.1759563Z echo "instance-type: $(get_ec2_metadata instance-type)"
2022-05-18T03:31:41.1759811Z echo "system info $(uname -a)"
2022-05-18T03:31:41.1770621Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-05-18T03:31:41.1770841Z env:
2022-05-18T03:31:41.1771000Z   IN_CI: 1
2022-05-18T03:31:41.1771148Z   IS_GHA: 1
2022-05-18T03:31:41.1771328Z   GIT_DEFAULT_BRANCH: master
2022-05-18T03:31:41.1771516Z ##[endgroup]
2022-05-18T03:31:41.1844524Z ami-id: ami-096198a0bccc6bad4
2022-05-18T03:31:41.1892507Z instance-id: i-0244965c0907218b7
2022-05-18T03:31:41.1938014Z instance-type: c5.2xlarge
2022-05-18T03:31:41.1944711Z system info Linux ip-10-0-5-106.ec2.internal 4.14.252-195.483.amzn2.x86_64 #1 SMP Mon Nov 1 20:58:46 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
2022-05-18T03:31:41.1958011Z ##[group]Run if systemctl is-active --quiet docker; then
2022-05-18T03:31:41.1958301Z if systemctl is-active --quiet docker; then
2022-05-18T03:31:41.1958547Z     echo "Docker daemon is running...";
2022-05-18T03:31:41.1958747Z else
2022-05-18T03:31:41.1958965Z     echo "Starting docker deamon..." && sudo systemctl start docker;
2022-05-18T03:31:41.1959187Z fi
2022-05-18T03:31:41.1969592Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-05-18T03:31:41.1969794Z env:
2022-05-18T03:31:41.1969948Z   IN_CI: 1
2022-05-18T03:31:41.1970110Z   IS_GHA: 1
2022-05-18T03:31:41.1970276Z   GIT_DEFAULT_BRANCH: master
2022-05-18T03:31:41.1970460Z ##[endgroup]
2022-05-18T03:31:41.2069590Z Docker daemon is running...
2022-05-18T03:31:41.2082665Z ##[group]Run AWS_ACCOUNT_ID=$(aws sts get-caller-identity|grep Account|cut -f4 -d\")
2022-05-18T03:31:41.2083019Z AWS_ACCOUNT_ID=$(aws sts get-caller-identity|grep Account|cut -f4 -d\")
2022-05-18T03:31:41.2083310Z retry () { "$@"  || (sleep 1 && "$@") || (sleep 2 && "$@") }
2022-05-18T03:31:41.2083758Z retry aws ecr get-login*** "$AWS_DEFAULT_REGION" | docker login --username AWS \
2022-05-18T03:31:41.2084089Z     --password-stdin "$AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com"
2022-05-18T03:31:41.2094105Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-05-18T03:31:41.2094325Z env:
2022-05-18T03:31:41.2094468Z   IN_CI: 1
2022-05-18T03:31:41.2094629Z   IS_GHA: 1
2022-05-18T03:31:41.2094811Z   GIT_DEFAULT_BRANCH: master
2022-05-18T03:31:41.2094993Z   AWS_RETRY_MODE: standard
2022-05-18T03:31:41.2095180Z   AWS_MAX_ATTEMPTS: 5
2022-05-18T03:31:41.2095377Z   AWS_DEFAULT_REGION: us-east-1
2022-05-18T03:31:41.2095556Z ##[endgroup]
2022-05-18T03:31:42.5404488Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json.
2022-05-18T03:31:42.5404978Z Configure a credential helper to remove this warning. See
2022-05-18T03:31:42.5405825Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store
2022-05-18T03:31:42.5406203Z 
2022-05-18T03:31:42.5406296Z Login Succeeded
2022-05-18T03:31:42.5435097Z ##[group]Run env | grep '^GITHUB' > "/tmp/github_env_${GITHUB_RUN_ID}"
2022-05-18T03:31:42.5435376Z env | grep '^GITHUB' > "/tmp/github_env_${GITHUB_RUN_ID}"
2022-05-18T03:31:42.5446337Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-05-18T03:31:42.5446558Z env:
2022-05-18T03:31:42.5446702Z   IN_CI: 1
2022-05-18T03:31:42.5446866Z   IS_GHA: 1
2022-05-18T03:31:42.5447044Z   GIT_DEFAULT_BRANCH: master
2022-05-18T03:31:42.5447220Z ##[endgroup]
2022-05-18T03:31:42.5497527Z Prepare all required actions
2022-05-18T03:31:42.5497806Z Getting action download info
2022-05-18T03:31:42.6950693Z Download action repository 'seemethere/add-github-ssh-key@v1' (SHA:1ecffedb1e192a50aa67dba2f0e048e5d3bfa144)
2022-05-18T03:31:42.8020145Z ##[group]Run ./.github/actions/setup-ssh
2022-05-18T03:31:42.8020342Z with:
2022-05-18T03:31:42.8020648Z   github-secret: ***
2022-05-18T03:31:42.8020818Z env:
2022-05-18T03:31:42.8020968Z   IN_CI: 1
2022-05-18T03:31:42.8021114Z   IS_GHA: 1
2022-05-18T03:31:42.8021290Z   GIT_DEFAULT_BRANCH: master
2022-05-18T03:31:42.8021474Z ##[endgroup]
2022-05-18T03:31:42.8039384Z ##[group]Run seemethere/add-github-ssh-key@v1
2022-05-18T03:31:42.8039591Z with:
2022-05-18T03:31:42.8039875Z   GITHUB_TOKEN: ***
2022-05-18T03:31:42.8040070Z   activate-with-label: false
2022-05-18T03:31:42.8040249Z   label: with-ssh
2022-05-18T03:31:42.8040440Z   remove-existing-keys: true
2022-05-18T03:31:42.8040621Z env:
2022-05-18T03:31:42.8040763Z   IN_CI: 1
2022-05-18T03:31:42.8040949Z   IS_GHA: 1
2022-05-18T03:31:42.8041110Z   GIT_DEFAULT_BRANCH: master
2022-05-18T03:31:42.8041297Z ##[endgroup]
2022-05-18T03:31:42.8533063Z Not on pull request and ciflow reference could not be extracted, skipping adding ssh keys
2022-05-18T03:31:42.8569828Z Prepare all required actions
2022-05-18T03:31:42.8585095Z ##[group]Run ./.github/actions/pull-docker-image
2022-05-18T03:31:42.8585301Z with:
2022-05-18T03:31:42.8585642Z   docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-xenial-py3.7-gcc5.4:6deab82db6a72ca54cd3e3322ee4f13864536734
2022-05-18T03:31:42.8585959Z env:
2022-05-18T03:31:42.8586110Z   IN_CI: 1
2022-05-18T03:31:42.8586268Z   IS_GHA: 1
2022-05-18T03:31:42.8586433Z   GIT_DEFAULT_BRANCH: master
2022-05-18T03:31:42.8586614Z ##[endgroup]
2022-05-18T03:31:42.8597399Z ##[group]Run retry () { "$@"  || (sleep 1 && "$@") || (sleep 2 && "$@") }
2022-05-18T03:31:42.8597667Z retry () { "$@"  || (sleep 1 && "$@") || (sleep 2 && "$@") }
2022-05-18T03:31:42.8597909Z retry docker pull "${DOCKER_IMAGE}"
2022-05-18T03:31:42.8608700Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-05-18T03:31:42.8609007Z env:
2022-05-18T03:31:42.8609161Z   IN_CI: 1
2022-05-18T03:31:42.8609339Z   IS_GHA: 1
2022-05-18T03:31:42.8609517Z   GIT_DEFAULT_BRANCH: master
2022-05-18T03:31:42.8609865Z   DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-xenial-py3.7-gcc5.4:6deab82db6a72ca54cd3e3322ee4f13864536734
2022-05-18T03:31:42.8610201Z ##[endgroup]
2022-05-18T03:31:43.1044136Z 6deab82db6a72ca54cd3e3322ee4f13864536734: Pulling from pytorch/pytorch-linux-xenial-py3.7-gcc5.4
2022-05-18T03:31:43.1051566Z 58690f9b18fc: Pulling fs layer
2022-05-18T03:31:43.1053027Z b51569e7c507: Pulling fs layer
2022-05-18T03:31:43.1053238Z da8ef40b9eca: Pulling fs layer
2022-05-18T03:31:43.1053443Z fb15d46c38dc: Pulling fs layer
2022-05-18T03:31:43.1053629Z 5ba54a79e67d: Pulling fs layer
2022-05-18T03:31:43.1053830Z 0a20b8d84c46: Pulling fs layer
2022-05-18T03:31:43.1054021Z 5877c23144ae: Pulling fs layer
2022-05-18T03:31:43.1054199Z d3e83054f718: Pulling fs layer
2022-05-18T03:31:43.1054404Z 140e7e919a6a: Pulling fs layer
2022-05-18T03:31:43.1054599Z 00fe5dff19d6: Pulling fs layer
2022-05-18T03:31:43.1057436Z 5253901bce0a: Pulling fs layer
2022-05-18T03:31:43.1057795Z f2ad3e4779d8: Pulling fs layer
2022-05-18T03:31:43.1058116Z d1935ca92dc4: Pulling fs layer
2022-05-18T03:31:43.1058422Z 370a68d9a452: Pulling fs layer
2022-05-18T03:31:43.1058723Z 5ba54a79e67d: Waiting
2022-05-18T03:31:43.1059059Z 92c75209b8cf: Pulling fs layer
2022-05-18T03:31:43.1059414Z fdd1b5b4d4e2: Pulling fs layer
2022-05-18T03:31:43.1059714Z 641ed2a0ee80: Pulling fs layer
2022-05-18T03:31:43.1060027Z 0a20b8d84c46: Waiting
2022-05-18T03:31:43.1060342Z fb15d46c38dc: Waiting
2022-05-18T03:31:43.1060697Z 17ceb3758ec4: Pulling fs layer
2022-05-18T03:31:43.1061055Z 81ca05b8cd5a: Pulling fs layer
2022-05-18T03:31:43.1061389Z d3e83054f718: Waiting
2022-05-18T03:31:43.1061709Z d124e94e1971: Pulling fs layer
2022-05-18T03:31:43.1062020Z 140e7e919a6a: Waiting
2022-05-18T03:31:43.1062339Z 00fe5dff19d6: Waiting
2022-05-18T03:31:43.1062639Z d4bdbe109a27: Pulling fs layer
2022-05-18T03:31:43.1063083Z 070594bc61e9: Pulling fs layer
2022-05-18T03:31:43.1063426Z 92c75209b8cf: Waiting
2022-05-18T03:31:43.1063765Z 2c9ca9e145e6: Pulling fs layer
2022-05-18T03:31:43.1064061Z 5253901bce0a: Waiting
2022-05-18T03:31:43.1064244Z 9d031d383e17: Pulling fs layer
2022-05-18T03:31:43.1064437Z 62e4a89ba8d3: Pulling fs layer
2022-05-18T03:31:43.1064610Z fdd1b5b4d4e2: Waiting
2022-05-18T03:31:43.1064794Z 171756362e4e: Pulling fs layer
2022-05-18T03:31:43.1064976Z f2ad3e4779d8: Waiting
2022-05-18T03:31:43.1065135Z 641ed2a0ee80: Waiting
2022-05-18T03:31:43.1065316Z ea726e35e256: Pulling fs layer
2022-05-18T03:31:43.1065505Z 721b024a3031: Pulling fs layer
2022-05-18T03:31:43.1065680Z 206a6f5bfe61: Pulling fs layer
2022-05-18T03:31:43.1065912Z d1935ca92dc4: Waiting
2022-05-18T03:31:43.1066188Z 45b7b7460778: Pulling fs layer
2022-05-18T03:31:43.1066466Z 28ef77622cff: Pulling fs layer
2022-05-18T03:31:43.1066737Z 17ceb3758ec4: Waiting
2022-05-18T03:31:43.1067023Z 2c9ca9e145e6: Waiting
2022-05-18T03:31:43.1067299Z 070594bc61e9: Waiting
2022-05-18T03:31:43.1067762Z d4bdbe109a27: Waiting
2022-05-18T03:31:43.1068044Z ead995a9636d: Pulling fs layer
2022-05-18T03:31:43.1068317Z d124e94e1971: Waiting
2022-05-18T03:31:43.1068568Z 81ca05b8cd5a: Waiting
2022-05-18T03:31:43.1068750Z 55366a8087ad: Pulling fs layer
2022-05-18T03:31:43.1068928Z a01ab60b3807: Pulling fs layer
2022-05-18T03:31:43.1069104Z 171756362e4e: Waiting
2022-05-18T03:31:43.1141840Z 9d031d383e17: Waiting
2022-05-18T03:31:43.1142269Z c9a9d301cafd: Pulling fs layer
2022-05-18T03:31:43.1142570Z ea726e35e256: Waiting
2022-05-18T03:31:43.1143024Z 62e4a89ba8d3: Waiting
2022-05-18T03:31:43.1143354Z 275239b0f78d: Pulling fs layer
2022-05-18T03:31:43.1143592Z 206a6f5bfe61: Waiting
2022-05-18T03:31:43.1143779Z 3550d2a21107: Pulling fs layer
2022-05-18T03:31:43.1144041Z 45b7b7460778: Waiting
2022-05-18T03:31:43.1144387Z 586f2f9bc005: Pulling fs layer
2022-05-18T03:31:43.1144857Z 28ef77622cff: Waiting
2022-05-18T03:31:43.1145020Z c9a9d301cafd: Waiting
2022-05-18T03:31:43.1145194Z 721b024a3031: Waiting
2022-05-18T03:31:43.1145370Z ead995a9636d: Waiting
2022-05-18T03:31:43.1145614Z 11fd06f0243a: Pulling fs layer
2022-05-18T03:31:43.1145952Z 275239b0f78d: Waiting
2022-05-18T03:31:43.1146259Z 55366a8087ad: Waiting
2022-05-18T03:31:43.1146511Z 477485598060: Pulling fs layer
2022-05-18T03:31:43.1146711Z a01ab60b3807: Waiting
2022-05-18T03:31:43.1147019Z aaeef6a5d26a: Pulling fs layer
2022-05-18T03:31:43.1147350Z 3550d2a21107: Waiting
2022-05-18T03:31:43.1147624Z 586f2f9bc005: Waiting
2022-05-18T03:31:43.1147806Z e9b66d11d0f7: Pulling fs layer
2022-05-18T03:31:43.1147984Z 242315b336c5: Pulling fs layer
2022-05-18T03:31:43.1148155Z 477485598060: Waiting
2022-05-18T03:31:43.1148323Z e9b66d11d0f7: Waiting
2022-05-18T03:31:43.1148548Z 7e414c970966: Pulling fs layer
2022-05-18T03:31:43.1148763Z 242315b336c5: Waiting
2022-05-18T03:31:43.1148978Z 81551a9ff750: Pulling fs layer
2022-05-18T03:31:43.1149163Z e673be102bed: Pulling fs layer
2022-05-18T03:31:43.1149347Z 7e414c970966: Waiting
2022-05-18T03:31:43.1149583Z 4f94094bd9bd: Pulling fs layer
2022-05-18T03:31:43.1149776Z 98756dfdd888: Pulling fs layer
2022-05-18T03:31:43.1149952Z 81551a9ff750: Waiting
2022-05-18T03:31:43.1150132Z 1238debabecc: Pulling fs layer
2022-05-18T03:31:43.1150330Z cefaab4f809a: Pulling fs layer
2022-05-18T03:31:43.1150517Z ebced6807dae: Pulling fs layer
2022-05-18T03:31:43.1150709Z 912549afe5e1: Pulling fs layer
2022-05-18T03:31:43.1150928Z e673be102bed: Waiting
2022-05-18T03:31:43.1151089Z 4f94094bd9bd: Waiting
2022-05-18T03:31:43.1151276Z 215c8a788eb9: Pulling fs layer
2022-05-18T03:31:43.1151534Z 98756dfdd888: Waiting
2022-05-18T03:31:43.1151705Z 61717ae21dd2: Pulling fs layer
2022-05-18T03:31:43.1151891Z cefaab4f809a: Waiting
2022-05-18T03:31:43.1152080Z ebced6807dae: Waiting
2022-05-18T03:31:43.1152256Z 215c8a788eb9: Waiting
2022-05-18T03:31:43.1152432Z 61717ae21dd2: Waiting
2022-05-18T03:31:43.1727694Z da8ef40b9eca: Verifying Checksum
2022-05-18T03:31:43.1728113Z da8ef40b9eca: Download complete
2022-05-18T03:31:43.1781843Z b51569e7c507: Verifying Checksum
2022-05-18T03:31:43.1782132Z b51569e7c507: Download complete
2022-05-18T03:31:43.2534010Z fb15d46c38dc: Download complete
2022-05-18T03:31:43.2620586Z 5ba54a79e67d: Verifying Checksum
2022-05-18T03:31:43.2620965Z 5ba54a79e67d: Download complete
2022-05-18T03:31:43.3340643Z 5877c23144ae: Verifying Checksum
2022-05-18T03:31:43.3340890Z 5877c23144ae: Download complete
2022-05-18T03:31:43.6317931Z 58690f9b18fc: Verifying Checksum
2022-05-18T03:31:43.6318361Z 58690f9b18fc: Download complete
2022-05-18T03:31:43.6466963Z d3e83054f718: Download complete
2022-05-18T03:31:43.6956921Z 140e7e919a6a: Download complete
2022-05-18T03:31:43.7216206Z 00fe5dff19d6: Download complete
2022-05-18T03:31:43.7765436Z 5253901bce0a: Download complete
2022-05-18T03:31:43.8048732Z f2ad3e4779d8: Download complete
2022-05-18T03:31:43.8407488Z d1935ca92dc4: Verifying Checksum
2022-05-18T03:31:43.8407884Z d1935ca92dc4: Download complete
2022-05-18T03:31:43.9245423Z 92c75209b8cf: Download complete
2022-05-18T03:31:43.9992523Z fdd1b5b4d4e2: Verifying Checksum
2022-05-18T03:31:43.9993075Z fdd1b5b4d4e2: Download complete
2022-05-18T03:31:44.1999067Z 370a68d9a452: Verifying Checksum
2022-05-18T03:31:44.1999511Z 370a68d9a452: Download complete
2022-05-18T03:31:44.2758390Z 17ceb3758ec4: Download complete
2022-05-18T03:31:44.3481196Z 81ca05b8cd5a: Verifying Checksum
2022-05-18T03:31:44.3481636Z 81ca05b8cd5a: Download complete
2022-05-18T03:31:44.4252524Z d124e94e1971: Verifying Checksum
2022-05-18T03:31:44.4252883Z d124e94e1971: Download complete
2022-05-18T03:31:44.4969370Z d4bdbe109a27: Verifying Checksum
2022-05-18T03:31:44.4970095Z d4bdbe109a27: Download complete
2022-05-18T03:31:44.5939801Z 070594bc61e9: Verifying Checksum
2022-05-18T03:31:44.5940213Z 070594bc61e9: Download complete
2022-05-18T03:31:44.6797079Z 2c9ca9e145e6: Verifying Checksum
2022-05-18T03:31:44.6797535Z 2c9ca9e145e6: Download complete
2022-05-18T03:31:44.7920036Z 58690f9b18fc: Pull complete
2022-05-18T03:31:44.9007668Z b51569e7c507: Pull complete
2022-05-18T03:31:45.0378525Z da8ef40b9eca: Pull complete
2022-05-18T03:31:45.1783616Z fb15d46c38dc: Pull complete
2022-05-18T03:31:45.3767601Z 5ba54a79e67d: Pull complete
2022-05-18T03:31:45.5148336Z 9d031d383e17: Verifying Checksum
2022-05-18T03:31:45.5148788Z 9d031d383e17: Download complete
2022-05-18T03:31:45.6566012Z 62e4a89ba8d3: Verifying Checksum
2022-05-18T03:31:45.6566366Z 62e4a89ba8d3: Download complete
2022-05-18T03:31:45.6786055Z 0a20b8d84c46: Download complete
2022-05-18T03:31:45.7546735Z 171756362e4e: Verifying Checksum
2022-05-18T03:31:45.7547110Z 171756362e4e: Download complete
2022-05-18T03:31:45.7664576Z ea726e35e256: Verifying Checksum
2022-05-18T03:31:45.7665109Z ea726e35e256: Download complete
2022-05-18T03:31:45.8345340Z 721b024a3031: Download complete
2022-05-18T03:31:45.8506369Z 206a6f5bfe61: Download complete
2022-05-18T03:31:45.9246839Z 28ef77622cff: Verifying Checksum
2022-05-18T03:31:45.9247227Z 28ef77622cff: Download complete
2022-05-18T03:31:46.0006934Z ead995a9636d: Verifying Checksum
2022-05-18T03:31:46.0007862Z ead995a9636d: Download complete
2022-05-18T03:31:46.0798790Z 55366a8087ad: Verifying Checksum
2022-05-18T03:31:46.0799311Z 55366a8087ad: Download complete
2022-05-18T03:31:46.1487970Z a01ab60b3807: Verifying Checksum
2022-05-18T03:31:46.1488599Z a01ab60b3807: Download complete
2022-05-18T03:31:46.2400485Z c9a9d301cafd: Verifying Checksum
2022-05-18T03:31:46.2401123Z c9a9d301cafd: Download complete
2022-05-18T03:31:46.3249473Z 275239b0f78d: Verifying Checksum
2022-05-18T03:31:46.3249904Z 275239b0f78d: Download complete
2022-05-18T03:31:46.4019399Z 3550d2a21107: Verifying Checksum
2022-05-18T03:31:46.4020204Z 3550d2a21107: Download complete
2022-05-18T03:31:46.4755896Z 586f2f9bc005: Download complete
2022-05-18T03:31:46.5491941Z 11fd06f0243a: Verifying Checksum
2022-05-18T03:31:46.5492326Z 11fd06f0243a: Download complete
2022-05-18T03:31:46.6241491Z 477485598060: Verifying Checksum
2022-05-18T03:31:46.6242193Z 477485598060: Download complete
2022-05-18T03:31:46.7056930Z aaeef6a5d26a: Verifying Checksum
2022-05-18T03:31:46.7057428Z aaeef6a5d26a: Download complete
2022-05-18T03:31:46.7733388Z e9b66d11d0f7: Verifying Checksum
2022-05-18T03:31:46.7733633Z e9b66d11d0f7: Download complete
2022-05-18T03:31:46.8555630Z 45b7b7460778: Verifying Checksum
2022-05-18T03:31:46.8556072Z 45b7b7460778: Download complete
2022-05-18T03:31:46.9238350Z 7e414c970966: Verifying Checksum
2022-05-18T03:31:46.9238714Z 7e414c970966: Download complete
2022-05-18T03:31:46.9984008Z 81551a9ff750: Verifying Checksum
2022-05-18T03:31:46.9985162Z 81551a9ff750: Download complete
2022-05-18T03:31:47.0871933Z e673be102bed: Verifying Checksum
2022-05-18T03:31:47.0872483Z e673be102bed: Download complete
2022-05-18T03:31:47.1576255Z 4f94094bd9bd: Verifying Checksum
2022-05-18T03:31:47.1576891Z 4f94094bd9bd: Download complete
2022-05-18T03:31:47.1921082Z 242315b336c5: Verifying Checksum
2022-05-18T03:31:47.1921577Z 242315b336c5: Download complete
2022-05-18T03:31:47.2667745Z 1238debabecc: Verifying Checksum
2022-05-18T03:31:47.2668490Z 1238debabecc: Download complete
2022-05-18T03:31:47.4023418Z 98756dfdd888: Download complete
2022-05-18T03:31:47.4763647Z ebced6807dae: Verifying Checksum
2022-05-18T03:31:47.4764690Z ebced6807dae: Download complete
2022-05-18T03:31:47.5845113Z 912549afe5e1: Verifying Checksum
2022-05-18T03:31:47.5846228Z 912549afe5e1: Download complete
2022-05-18T03:31:47.6686442Z 215c8a788eb9: Download complete
2022-05-18T03:31:48.2619923Z 61717ae21dd2: Verifying Checksum
2022-05-18T03:31:48.2620200Z 61717ae21dd2: Download complete
2022-05-18T03:31:50.9068331Z cefaab4f809a: Verifying Checksum
2022-05-18T03:31:50.9069090Z cefaab4f809a: Download complete
2022-05-18T03:31:51.0856161Z 0a20b8d84c46: Pull complete
2022-05-18T03:31:51.2747875Z 5877c23144ae: Pull complete
2022-05-18T03:31:51.4991469Z d3e83054f718: Pull complete
2022-05-18T03:31:51.7415698Z 140e7e919a6a: Pull complete
2022-05-18T03:31:51.9997208Z 00fe5dff19d6: Pull complete
2022-05-18T03:31:52.2222716Z 5253901bce0a: Pull complete
2022-05-18T03:31:52.4679134Z f2ad3e4779d8: Pull complete
2022-05-18T03:31:52.6900213Z d1935ca92dc4: Pull complete
2022-05-18T03:31:53.8738164Z 370a68d9a452: Pull complete
2022-05-18T03:31:54.1234217Z 92c75209b8cf: Pull complete
2022-05-18T03:31:54.2403039Z fdd1b5b4d4e2: Pull complete
2022-05-18T03:31:56.5353750Z 641ed2a0ee80: Verifying Checksum
2022-05-18T03:31:56.5354090Z 641ed2a0ee80: Download complete
2022-05-18T03:32:14.7038368Z 641ed2a0ee80: Pull complete
2022-05-18T03:32:15.0030821Z 17ceb3758ec4: Pull complete
2022-05-18T03:32:15.2719274Z 81ca05b8cd5a: Pull complete
2022-05-18T03:32:15.3956890Z d124e94e1971: Pull complete
2022-05-18T03:32:15.5650023Z d4bdbe109a27: Pull complete
2022-05-18T03:32:15.7502581Z 070594bc61e9: Pull complete
2022-05-18T03:32:15.8457536Z 2c9ca9e145e6: Pull complete
2022-05-18T03:32:17.5091941Z 9d031d383e17: Pull complete
2022-05-18T03:32:17.7224727Z 62e4a89ba8d3: Pull complete
2022-05-18T03:32:17.9047823Z 171756362e4e: Pull complete
2022-05-18T03:32:18.1531640Z ea726e35e256: Pull complete
2022-05-18T03:32:18.3874064Z 721b024a3031: Pull complete
2022-05-18T03:32:18.5842827Z 206a6f5bfe61: Pull complete
2022-05-18T03:32:20.6473854Z 45b7b7460778: Pull complete
2022-05-18T03:32:20.8765813Z 28ef77622cff: Pull complete
2022-05-18T03:32:21.1311405Z ead995a9636d: Pull complete
2022-05-18T03:32:21.3097824Z 55366a8087ad: Pull complete
2022-05-18T03:32:21.5221895Z a01ab60b3807: Pull complete
2022-05-18T03:32:21.6747096Z c9a9d301cafd: Pull complete
2022-05-18T03:32:21.8306685Z 275239b0f78d: Pull complete
2022-05-18T03:32:22.0719247Z 3550d2a21107: Pull complete
2022-05-18T03:32:22.2922776Z 586f2f9bc005: Pull complete
2022-05-18T03:32:22.5374981Z 11fd06f0243a: Pull complete
2022-05-18T03:32:22.7841516Z 477485598060: Pull complete
2022-05-18T03:32:23.0104219Z aaeef6a5d26a: Pull complete
2022-05-18T03:32:23.2588961Z e9b66d11d0f7: Pull complete
2022-05-18T03:32:24.3256089Z 242315b336c5: Pull complete
2022-05-18T03:32:24.4216919Z 7e414c970966: Pull complete
2022-05-18T03:32:24.5099350Z 81551a9ff750: Pull complete
2022-05-18T03:32:24.6217134Z e673be102bed: Pull complete
2022-05-18T03:32:24.7307805Z 4f94094bd9bd: Pull complete
2022-05-18T03:32:25.0238004Z 98756dfdd888: Pull complete
2022-05-18T03:32:25.1296052Z 1238debabecc: Pull complete
2022-05-18T03:32:29.5991855Z cefaab4f809a: Pull complete
2022-05-18T03:32:29.7518564Z ebced6807dae: Pull complete
2022-05-18T03:32:29.8412946Z 912549afe5e1: Pull complete
2022-05-18T03:32:29.9289527Z 215c8a788eb9: Pull complete
2022-05-18T03:32:31.5134295Z 61717ae21dd2: Pull complete
2022-05-18T03:32:31.6240879Z Digest: sha256:9c228d64aeaa1a84153f684d8bf8d2b818b53df05ec50809bfb8bb625f2aea5c
2022-05-18T03:32:31.6642280Z Status: Downloaded newer image for 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-xenial-py3.7-gcc5.4:6deab82db6a72ca54cd3e3322ee4f13864536734
2022-05-18T03:32:31.6883520Z 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-xenial-py3.7-gcc5.4:6deab82db6a72ca54cd3e3322ee4f13864536734
2022-05-18T03:32:31.6952831Z Prepare all required actions
2022-05-18T03:32:31.6953072Z Getting action download info
2022-05-18T03:32:31.8502770Z Download action repository 'seemethere/download-artifact-s3@v3' (SHA:64048a097659c8ca71ceacbb3c01cee9ed6f1b05)
2022-05-18T03:32:31.9972639Z Download action repository 'actions/download-artifact@v2' (SHA:f023be2c48cc18debc3bacd34cb396e0295e2869)
2022-05-18T03:32:32.1052950Z ##[group]Run ./.github/actions/download-build-artifacts
2022-05-18T03:32:32.1053177Z with:
2022-05-18T03:32:32.1053356Z   name: linux-xenial-py3.7-gcc5.4
2022-05-18T03:32:32.1053545Z env:
2022-05-18T03:32:32.1053701Z   IN_CI: 1
2022-05-18T03:32:32.1054087Z   IS_GHA: 1
2022-05-18T03:32:32.1054266Z   GIT_DEFAULT_BRANCH: master
2022-05-18T03:32:32.1054453Z ##[endgroup]
2022-05-18T03:32:32.1076883Z ##[group]Run seemethere/download-artifact-s3@v3
2022-05-18T03:32:32.1077099Z with:
2022-05-18T03:32:32.1077276Z   name: linux-xenial-py3.7-gcc5.4
2022-05-18T03:32:32.1077597Z   s3-bucket: gha-artifacts
2022-05-18T03:32:32.1077821Z   region: us-east-1
2022-05-18T03:32:32.1077976Z env:
2022-05-18T03:32:32.1078129Z   IN_CI: 1
2022-05-18T03:32:32.1078289Z   IS_GHA: 1
2022-05-18T03:32:32.1078454Z   GIT_DEFAULT_BRANCH: master
2022-05-18T03:32:32.1078821Z ##[endgroup]
2022-05-18T03:32:32.5043914Z Found 1 objects with prefix pytorch/pytorch/2342799944/1/linux-xenial-py3.7-gcc5.4/
2022-05-18T03:32:32.5044385Z Starting download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip
2022-05-18T03:32:37.0191717Z Finished download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip
2022-05-18T03:32:37.0192114Z 
2022-05-18T03:32:37.0199991Z Artifact download has finished successfully
2022-05-18T03:32:37.0286857Z ##[group]Run unzip -o artifacts.zip
2022-05-18T03:32:37.0287084Z unzip -o artifacts.zip
2022-05-18T03:32:37.0297786Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-05-18T03:32:37.0298001Z env:
2022-05-18T03:32:37.0298164Z   IN_CI: 1
2022-05-18T03:32:37.0298318Z   IS_GHA: 1
2022-05-18T03:32:37.0298500Z   GIT_DEFAULT_BRANCH: master
2022-05-18T03:32:37.0298689Z ##[endgroup]
2022-05-18T03:32:37.0603506Z Archive:  artifacts.zip
2022-05-18T03:32:37.0604485Z    creating: dist/
2022-05-18T03:32:37.7560440Z   inflating: dist/torch-1.12.0a0+git3b23752-cp37-cp37m-linux_x86_64.whl  
2022-05-18T03:32:37.7560980Z    creating: build/custom_test_artifacts/
2022-05-18T03:32:37.7561297Z    creating: build/custom_test_artifacts/custom-op-build/
2022-05-18T03:32:37.7561640Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/
2022-05-18T03:32:37.7563354Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeOutput.log  
2022-05-18T03:32:37.7564093Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.12.4/
2022-05-18T03:32:37.7564820Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.12.4/CMakeSystem.cmake  
2022-05-18T03:32:37.7565252Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.12.4/CompilerIdC/
2022-05-18T03:32:37.7565670Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.12.4/CompilerIdC/tmp/
2022-05-18T03:32:37.7566513Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.12.4/CompilerIdC/CMakeCCompilerId.c  
2022-05-18T03:32:37.7567299Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.12.4/CompilerIdC/a.out  
2022-05-18T03:32:37.7568013Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.12.4/CompilerIdCXX/
2022-05-18T03:32:37.7568437Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.12.4/CompilerIdCXX/tmp/
2022-05-18T03:32:37.7569299Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.12.4/CompilerIdCXX/CMakeCXXCompilerId.cpp  
2022-05-18T03:32:37.7570293Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.12.4/CompilerIdCXX/a.out  
2022-05-18T03:32:37.7571698Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.12.4/CMakeDetermineCompilerABI_C.bin  
2022-05-18T03:32:37.7572447Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.12.4/CMakeCCompiler.cmake  
2022-05-18T03:32:37.7573237Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.12.4/CMakeDetermineCompilerABI_CXX.bin  
2022-05-18T03:32:37.7574073Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.12.4/CMakeCXXCompiler.cmake  
2022-05-18T03:32:37.7574771Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeTmp/
2022-05-18T03:32:37.7575236Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/feature_tests.c  
2022-05-18T03:32:37.7575640Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/feature_tests.cxx  
2022-05-18T03:32:37.7576165Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/feature_tests.bin  
2022-05-18T03:32:37.7577057Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeError.log  
2022-05-18T03:32:37.7577689Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/cmake.check_cache  
2022-05-18T03:32:37.7578105Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/
2022-05-18T03:32:37.7595377Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/depend.make  
2022-05-18T03:32:37.7595825Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/link.txt  
2022-05-18T03:32:37.7596255Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/cmake_clean.cmake  
2022-05-18T03:32:37.7596774Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/build.make  
2022-05-18T03:32:37.7597226Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/DependInfo.cmake  
2022-05-18T03:32:37.7597673Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/flags.make  
2022-05-18T03:32:37.7598108Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/progress.make  
2022-05-18T03:32:37.7640580Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/CXX.includecache  
2022-05-18T03:32:37.7654077Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/depend.internal  
2022-05-18T03:32:37.7738803Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o  
2022-05-18T03:32:37.7739235Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/
2022-05-18T03:32:37.7758980Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/depend.make  
2022-05-18T03:32:37.7759462Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/link.txt  
2022-05-18T03:32:37.7759925Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/cmake_clean.cmake  
2022-05-18T03:32:37.7760463Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/build.make  
2022-05-18T03:32:37.7760928Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/DependInfo.cmake  
2022-05-18T03:32:37.7761390Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/flags.make  
2022-05-18T03:32:37.7761841Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/progress.make  
2022-05-18T03:32:37.7804203Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/CXX.includecache  
2022-05-18T03:32:37.7817636Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/depend.internal  
2022-05-18T03:32:37.7878828Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o  
2022-05-18T03:32:37.7879311Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeDirectoryInformation.cmake  
2022-05-18T03:32:37.7879911Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/TargetDirectories.txt  
2022-05-18T03:32:37.7880344Z  extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/progress.marks  
2022-05-18T03:32:37.7880755Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile2  
2022-05-18T03:32:37.7881599Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile.cmake  
2022-05-18T03:32:37.7882950Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeCache.txt  
2022-05-18T03:32:37.7883595Z   inflating: build/custom_test_artifacts/custom-op-build/Makefile  
2022-05-18T03:32:37.7884242Z   inflating: build/custom_test_artifacts/custom-op-build/cmake_install.cmake  
2022-05-18T03:32:37.7956962Z   inflating: build/custom_test_artifacts/custom-op-build/libcustom_ops.so  
2022-05-18T03:32:37.8004759Z   inflating: build/custom_test_artifacts/custom-op-build/test_custom_ops  
2022-05-18T03:32:37.8005554Z    creating: build/custom_test_artifacts/jit-hook-build/
2022-05-18T03:32:37.8005885Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/
2022-05-18T03:32:37.8007915Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeOutput.log  
2022-05-18T03:32:37.8008643Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.12.4/
2022-05-18T03:32:37.8009344Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.12.4/CMakeSystem.cmake  
2022-05-18T03:32:37.8009764Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.12.4/CompilerIdC/
2022-05-18T03:32:37.8010239Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.12.4/CompilerIdC/tmp/
2022-05-18T03:32:37.8011058Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.12.4/CompilerIdC/CMakeCCompilerId.c  
2022-05-18T03:32:37.8011735Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.12.4/CompilerIdC/a.out  
2022-05-18T03:32:37.8012473Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.12.4/CompilerIdCXX/
2022-05-18T03:32:37.8012899Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.12.4/CompilerIdCXX/tmp/
2022-05-18T03:32:37.8013863Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.12.4/CompilerIdCXX/CMakeCXXCompilerId.cpp  
2022-05-18T03:32:37.8014805Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.12.4/CompilerIdCXX/a.out  
2022-05-18T03:32:37.8015934Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.12.4/CMakeDetermineCompilerABI_C.bin  
2022-05-18T03:32:37.8016777Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.12.4/CMakeCCompiler.cmake  
2022-05-18T03:32:37.8017600Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.12.4/CMakeDetermineCompilerABI_CXX.bin  
2022-05-18T03:32:37.8018418Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.12.4/CMakeCXXCompiler.cmake  
2022-05-18T03:32:37.8019195Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeTmp/
2022-05-18T03:32:37.8019934Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/feature_tests.c  
2022-05-18T03:32:37.8020344Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/feature_tests.cxx  
2022-05-18T03:32:37.8021013Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/feature_tests.bin  
2022-05-18T03:32:37.8021739Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeError.log  
2022-05-18T03:32:37.8022411Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/cmake.check_cache  
2022-05-18T03:32:37.8022817Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/
2022-05-18T03:32:37.8041934Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/depend.make  
2022-05-18T03:32:37.8042779Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/link.txt  
2022-05-18T03:32:37.8043744Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/cmake_clean.cmake  
2022-05-18T03:32:37.8044597Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/build.make  
2022-05-18T03:32:37.8045156Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/DependInfo.cmake  
2022-05-18T03:32:37.8045603Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/flags.make  
2022-05-18T03:32:37.8046046Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/progress.make  
2022-05-18T03:32:37.8087139Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/CXX.includecache  
2022-05-18T03:32:37.8100340Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/depend.internal  
2022-05-18T03:32:37.8148300Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o  
2022-05-18T03:32:37.8149304Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeDirectoryInformation.cmake  
2022-05-18T03:32:37.8150154Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/TargetDirectories.txt  
2022-05-18T03:32:37.8150970Z  extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/progress.marks  
2022-05-18T03:32:37.8151523Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile2  
2022-05-18T03:32:37.8152122Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile.cmake  
2022-05-18T03:32:37.8152832Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeCache.txt  
2022-05-18T03:32:37.8153491Z   inflating: build/custom_test_artifacts/jit-hook-build/Makefile  
2022-05-18T03:32:37.8153922Z   inflating: build/custom_test_artifacts/jit-hook-build/cmake_install.cmake  
2022-05-18T03:32:37.8191500Z   inflating: build/custom_test_artifacts/jit-hook-build/test_jit_hooks  
2022-05-18T03:32:37.8192185Z    creating: build/custom_test_artifacts/custom-backend-build/
2022-05-18T03:32:37.8192564Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/
2022-05-18T03:32:37.8194768Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeOutput.log  
2022-05-18T03:32:37.8195551Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.12.4/
2022-05-18T03:32:37.8196197Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.12.4/CMakeSystem.cmake  
2022-05-18T03:32:37.8196635Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.12.4/CompilerIdC/
2022-05-18T03:32:37.8197291Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.12.4/CompilerIdC/tmp/
2022-05-18T03:32:37.8198015Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.12.4/CompilerIdC/CMakeCCompilerId.c  
2022-05-18T03:32:37.8198866Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.12.4/CompilerIdC/a.out  
2022-05-18T03:32:37.8199399Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.12.4/CompilerIdCXX/
2022-05-18T03:32:37.8199860Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.12.4/CompilerIdCXX/tmp/
2022-05-18T03:32:37.8200766Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.12.4/CompilerIdCXX/CMakeCXXCompilerId.cpp  
2022-05-18T03:32:37.8201565Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.12.4/CompilerIdCXX/a.out  
2022-05-18T03:32:37.8202507Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.12.4/CMakeDetermineCompilerABI_C.bin  
2022-05-18T03:32:37.8203331Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.12.4/CMakeCCompiler.cmake  
2022-05-18T03:32:37.8204148Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.12.4/CMakeDetermineCompilerABI_CXX.bin  
2022-05-18T03:32:37.8205166Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.12.4/CMakeCXXCompiler.cmake  
2022-05-18T03:32:37.8205950Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeTmp/
2022-05-18T03:32:37.8206573Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/feature_tests.c  
2022-05-18T03:32:37.8207114Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/feature_tests.cxx  
2022-05-18T03:32:37.8207892Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/feature_tests.bin  
2022-05-18T03:32:37.8208665Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeError.log  
2022-05-18T03:32:37.8209108Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/cmake.check_cache  
2022-05-18T03:32:37.8209540Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/
2022-05-18T03:32:37.8229484Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/depend.make  
2022-05-18T03:32:37.8230476Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/link.txt  
2022-05-18T03:32:37.8231415Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/cmake_clean.cmake  
2022-05-18T03:32:37.8232334Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/build.make  
2022-05-18T03:32:37.8232921Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/DependInfo.cmake  
2022-05-18T03:32:37.8233403Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/flags.make  
2022-05-18T03:32:37.8233886Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/progress.make  
2022-05-18T03:32:37.8274906Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/CXX.includecache  
2022-05-18T03:32:37.8288187Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/depend.internal  
2022-05-18T03:32:37.8331102Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o  
2022-05-18T03:32:37.8331739Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/
2022-05-18T03:32:37.8334865Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/depend.make  
2022-05-18T03:32:37.8335735Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/link.txt  
2022-05-18T03:32:37.8336575Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/cmake_clean.cmake  
2022-05-18T03:32:37.8337442Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/build.make  
2022-05-18T03:32:37.8338152Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/DependInfo.cmake  
2022-05-18T03:32:37.8338624Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/flags.make  
2022-05-18T03:32:37.8339085Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/progress.make  
2022-05-18T03:32:37.8342716Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/CXX.includecache  
2022-05-18T03:32:37.8345333Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/depend.internal  
2022-05-18T03:32:37.8456087Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o  
2022-05-18T03:32:37.8456993Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeDirectoryInformation.cmake  
2022-05-18T03:32:37.8457865Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/TargetDirectories.txt  
2022-05-18T03:32:37.8458713Z  extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/progress.marks  
2022-05-18T03:32:37.8459372Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile2  
2022-05-18T03:32:37.8459877Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile.cmake  
2022-05-18T03:32:37.8460627Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeCache.txt  
2022-05-18T03:32:37.8461347Z   inflating: build/custom_test_artifacts/custom-backend-build/Makefile  
2022-05-18T03:32:37.8461908Z   inflating: build/custom_test_artifacts/custom-backend-build/cmake_install.cmake  
2022-05-18T03:32:37.8554852Z   inflating: build/custom_test_artifacts/custom-backend-build/libcustom_backend.so  
2022-05-18T03:32:37.8589489Z   inflating: build/custom_test_artifacts/custom-backend-build/test_custom_backend  
2022-05-18T03:32:37.8589973Z    creating: build/lib/
2022-05-18T03:32:37.8590463Z   inflating: build/lib/libclog.a     
2022-05-18T03:32:37.8641865Z   inflating: build/lib/libgtest.a    
2022-05-18T03:32:37.8649594Z   inflating: build/lib/libpthreadpool.a  
2022-05-18T03:32:37.8715685Z   inflating: build/lib/libbenchmark.a  
2022-05-18T03:32:37.8800571Z   inflating: build/lib/libprotobuf-lite.a  
2022-05-18T03:32:37.8862065Z   inflating: build/lib/libasmjit.a   
2022-05-18T03:32:37.8887091Z   inflating: build/lib/libtensorpipe_uv.a  
2022-05-18T03:32:37.8979368Z   inflating: build/lib/libgloo.a     
2022-05-18T03:32:37.9397582Z   inflating: build/lib/libprotobuf.a  
2022-05-18T03:32:37.9413160Z   inflating: build/lib/libfmt.a      
2022-05-18T03:32:37.9413763Z   inflating: build/lib/libfoxi_loader.a  
2022-05-18T03:32:37.9414706Z   inflating: build/lib/libtorch_global_deps.so  
2022-05-18T03:32:37.9465531Z   inflating: build/lib/libc10.so     
2022-05-18T03:32:37.9473026Z   inflating: build/lib/libcpuinfo.a  
2022-05-18T03:32:37.9479798Z   inflating: build/lib/libcpuinfo_internals.a  
2022-05-18T03:32:37.9492441Z   inflating: build/lib/libqnnpack.a  
2022-05-18T03:32:37.9494286Z   inflating: build/lib/libnnpack_reference_layers.a  
2022-05-18T03:32:37.9513573Z   inflating: build/lib/libpytorch_qnnpack.a  
2022-05-18T03:32:37.9963839Z   inflating: build/lib/libprotoc.a   
2022-05-18T03:32:37.9978170Z   inflating: build/lib/libgmock.a    
2022-05-18T03:32:37.9978675Z   inflating: build/lib/libgtest_main.a  
2022-05-18T03:32:37.9979332Z   inflating: build/lib/libbenchmark_main.a  
2022-05-18T03:32:38.6599730Z   inflating: build/lib/libdnnl.a     
2022-05-18T03:32:38.6616097Z   inflating: build/lib/libnnpack.a   
2022-05-18T03:32:38.7165030Z   inflating: build/lib/libtensorpipe.a  
2022-05-18T03:32:38.7165543Z   inflating: build/lib/libgmock_main.a  
2022-05-18T03:32:38.8351452Z   inflating: build/lib/libfbgemm.a   
2022-05-18T03:32:38.9284606Z   inflating: build/lib/libdnnl_graph.a  
2022-05-18T03:32:38.9489533Z   inflating: build/lib/libkineto.a   
2022-05-18T03:32:38.9525889Z   inflating: build/lib/libcaffe2_protos.a  
2022-05-18T03:32:38.9637483Z   inflating: build/lib/libXNNPACK.a  
2022-05-18T03:32:38.9675581Z   inflating: build/lib/libonnx_proto.a  
2022-05-18T03:32:39.0209037Z   inflating: build/lib/libonnx.a     
2022-05-18T03:32:40.6667081Z   inflating: build/lib/libtorch_cpu.so  
2022-05-18T03:32:40.6667538Z   inflating: build/lib/libtorch.so   
2022-05-18T03:32:40.6686881Z   inflating: build/lib/libjitbackend_test.so  
2022-05-18T03:32:40.6711492Z   inflating: build/lib/libbackend_with_compiler.so  
2022-05-18T03:32:40.6753721Z   inflating: build/lib/libtorchbind_test.so  
2022-05-18T03:32:40.6756889Z   inflating: build/lib/libshm.so     
2022-05-18T03:32:40.8017769Z   inflating: build/lib/libtorch_python.so  
2022-05-18T03:32:40.8048277Z   inflating: build/lib/libnnapi_backend.so  
2022-05-18T03:32:40.8048518Z    creating: build/bin/
2022-05-18T03:32:40.8092769Z   inflating: build/bin/c10_registry_test  
2022-05-18T03:32:40.8152505Z   inflating: build/bin/c10_optional_test  
2022-05-18T03:32:40.8287514Z   inflating: build/bin/c10_intrusive_ptr_test  
2022-05-18T03:32:40.8327503Z   inflating: build/bin/c10_flags_test  
2022-05-18T03:32:40.8369851Z   inflating: build/bin/c10_exception_test  
2022-05-18T03:32:40.8415430Z   inflating: build/bin/c10_logging_test  
2022-05-18T03:32:40.8460165Z   inflating: build/bin/c10_complex_test  
2022-05-18T03:32:40.8546953Z   inflating: build/bin/c10_either_test  
2022-05-18T03:32:40.8587400Z   inflating: build/bin/c10_irange_test  
2022-05-18T03:32:40.8631871Z   inflating: build/bin/c10_bfloat16_test  
2022-05-18T03:32:40.8678523Z   inflating: build/bin/c10_string_view_test  
2022-05-18T03:32:40.8720181Z   inflating: build/bin/c10_accumulate_test  
2022-05-18T03:32:40.8763471Z   inflating: build/bin/c10_complex_math_test  
2022-05-18T03:32:40.8805736Z   inflating: build/bin/c10_Bitset_test  
2022-05-18T03:32:40.8917254Z   inflating: build/bin/c10_SmallVectorTest  
2022-05-18T03:32:40.8962103Z   inflating: build/bin/c10_typeid_test  
2022-05-18T03:32:40.9006291Z   inflating: build/bin/c10_InlineDeviceGuard_test  
2022-05-18T03:32:40.9051177Z   inflating: build/bin/c10_InlineStreamGuard_test  
2022-05-18T03:32:40.9090811Z   inflating: build/bin/c10_CompileTimeFunctionPointer_test  
2022-05-18T03:32:40.9131923Z   inflating: build/bin/c10_tempfile_test  
2022-05-18T03:32:40.9177215Z   inflating: build/bin/c10_SizesAndStrides_test  
2022-05-18T03:32:40.9215857Z   inflating: build/bin/c10_StreamGuard_test  
2022-05-18T03:32:40.9266406Z   inflating: build/bin/c10_ordered_preserving_dict_test  
2022-05-18T03:32:40.9311679Z   inflating: build/bin/c10_ThreadLocal_test  
2022-05-18T03:32:40.9357991Z   inflating: build/bin/c10_DispatchKeySet_test  
2022-05-18T03:32:40.9399419Z   inflating: build/bin/c10_DeviceGuard_test  
2022-05-18T03:32:40.9440253Z   inflating: build/bin/c10_C++17_test  
2022-05-18T03:32:40.9479117Z   inflating: build/bin/c10_TypeTraits_test  
2022-05-18T03:32:40.9519422Z   inflating: build/bin/c10_Device_test  
2022-05-18T03:32:40.9558918Z   inflating: build/bin/c10_DeadlockDetection_test  
2022-05-18T03:32:40.9598666Z   inflating: build/bin/c10_Half_test  
2022-05-18T03:32:40.9644916Z   inflating: build/bin/c10_LeftRight_test  
2022-05-18T03:32:40.9683550Z   inflating: build/bin/c10_ConstexprCrc_test  
2022-05-18T03:32:40.9733941Z   inflating: build/bin/c10_Metaprogramming_test  
2022-05-18T03:32:40.9772513Z   inflating: build/bin/c10_Array_test  
2022-05-18T03:32:40.9812491Z   inflating: build/bin/c10_Synchronized_test  
2022-05-18T03:32:40.9852942Z   inflating: build/bin/c10_TypeList_test  
2022-05-18T03:32:40.9894794Z   inflating: build/bin/c10_TypeIndex_test  
2022-05-18T03:32:40.9936407Z   inflating: build/bin/c10_intrusive_ptr_benchmark  
2022-05-18T03:32:41.0326631Z   inflating: build/bin/protoc-3.13.0.0  
2022-05-18T03:32:41.0716698Z   inflating: build/bin/protoc        
2022-05-18T03:32:41.0959832Z   inflating: build/bin/vec_test_all_types_DEFAULT  
2022-05-18T03:32:41.1226887Z   inflating: build/bin/vec_test_all_types_AVX2  
2022-05-18T03:32:41.1270310Z   inflating: build/bin/FileStoreTest  
2022-05-18T03:32:41.1313724Z   inflating: build/bin/HashStoreTest  
2022-05-18T03:32:41.1362041Z   inflating: build/bin/TCPStoreTest  
2022-05-18T03:32:41.1400782Z   inflating: build/bin/op_allowlist_test  
2022-05-18T03:32:41.1403059Z   inflating: build/bin/example_allreduce  
2022-05-18T03:32:41.1456859Z   inflating: build/bin/ProcessGroupGlooTest  
2022-05-18T03:32:41.1504807Z   inflating: build/bin/kernel_stackbased_test  
2022-05-18T03:32:41.1582020Z   inflating: build/bin/make_boxed_from_unboxed_functor_test  
2022-05-18T03:32:41.1659124Z   inflating: build/bin/kernel_function_test  
2022-05-18T03:32:41.1759427Z   inflating: build/bin/kernel_function_legacy_test  
2022-05-18T03:32:41.1803441Z   inflating: build/bin/backend_fallback_test  
2022-05-18T03:32:41.1855258Z   inflating: build/bin/KernelFunction_test  
2022-05-18T03:32:41.1903298Z   inflating: build/bin/IListRef_test  
2022-05-18T03:32:41.1944475Z   inflating: build/bin/stride_properties_test  
2022-05-18T03:32:41.1984569Z   inflating: build/bin/dispatch_key_set_test  
2022-05-18T03:32:41.2040059Z   inflating: build/bin/vmap_test     
2022-05-18T03:32:41.2089206Z   inflating: build/bin/type_test     
2022-05-18T03:32:41.2162526Z   inflating: build/bin/cpu_rng_test  
2022-05-18T03:32:41.2202213Z   inflating: build/bin/reduce_ops_test  
2022-05-18T03:32:41.2244800Z   inflating: build/bin/undefined_tensor_test  
2022-05-18T03:32:41.2323423Z   inflating: build/bin/ivalue_test   
2022-05-18T03:32:41.2371434Z   inflating: build/bin/apply_utils_test  
2022-05-18T03:32:41.2421119Z   inflating: build/bin/basic         
2022-05-18T03:32:41.2464731Z   inflating: build/bin/broadcast_test  
2022-05-18T03:32:41.2548195Z   inflating: build/bin/kernel_lambda_test  
2022-05-18T03:32:41.2595275Z   inflating: build/bin/cpu_generator_test  
2022-05-18T03:32:41.2825712Z   inflating: build/bin/op_registration_test  
2022-05-18T03:32:41.2870587Z   inflating: build/bin/half_test     
2022-05-18T03:32:41.2911633Z   inflating: build/bin/reportMemoryUsage_test  
2022-05-18T03:32:41.2953057Z   inflating: build/bin/Dimname_test  
2022-05-18T03:32:41.2994865Z   inflating: build/bin/memory_format_test  
2022-05-18T03:32:41.3039938Z   inflating: build/bin/test_parallel  
2022-05-18T03:32:41.3082424Z   inflating: build/bin/cpu_profiling_allocator_test  
2022-05-18T03:32:41.3185326Z   inflating: build/bin/kernel_lambda_legacy_test  
2022-05-18T03:32:41.3186297Z   inflating: build/bin/verify_api_visibility  
2022-05-18T03:32:41.3244615Z   inflating: build/bin/Dict_test     
2022-05-18T03:32:41.3290417Z   inflating: build/bin/scalar_test   
2022-05-18T03:32:41.3336032Z   inflating: build/bin/extension_backend_test  
2022-05-18T03:32:41.3378240Z   inflating: build/bin/inline_container_test  
2022-05-18T03:32:41.3466474Z   inflating: build/bin/List_test     
2022-05-18T03:32:41.3507661Z   inflating: build/bin/wrapdim_test  
2022-05-18T03:32:41.3553419Z   inflating: build/bin/native_test   
2022-05-18T03:32:41.3599313Z   inflating: build/bin/scalar_tensor_test  
2022-05-18T03:32:41.3638683Z   inflating: build/bin/lazy_tensor_test  
2022-05-18T03:32:41.3679833Z   inflating: build/bin/memory_overlapping_test  
2022-05-18T03:32:41.3727755Z   inflating: build/bin/atest         
2022-05-18T03:32:41.3773828Z   inflating: build/bin/quantized_test  
2022-05-18T03:32:41.3820102Z   inflating: build/bin/NamedTensor_test  
2022-05-18T03:32:41.3860238Z   inflating: build/bin/dlconvertor_test  
2022-05-18T03:32:41.3901637Z   inflating: build/bin/weakref_test  
2022-05-18T03:32:41.3903923Z   inflating: build/bin/thread_init_test  
2022-05-18T03:32:41.3944611Z   inflating: build/bin/operators_test  
2022-05-18T03:32:41.3985456Z   inflating: build/bin/CppSignature_test  
2022-05-18T03:32:41.4048086Z   inflating: build/bin/tensor_iterator_test  
2022-05-18T03:32:41.4088142Z   inflating: build/bin/variant_test  
2022-05-18T03:32:41.4131051Z   inflating: build/bin/math_kernel_test  
2022-05-18T03:32:41.4185290Z   inflating: build/bin/pow_test      
2022-05-18T03:32:41.4227632Z   inflating: build/bin/mobile_memory_cleanup  
2022-05-18T03:32:41.4241732Z   inflating: build/bin/tutorial_tensorexpr  
2022-05-18T03:32:41.4285769Z   inflating: build/bin/test_dist_autograd  
2022-05-18T03:32:41.4343267Z   inflating: build/bin/test_cpp_rpc  
2022-05-18T03:32:41.4345416Z   inflating: build/bin/parallel_benchmark  
2022-05-18T03:32:41.4401363Z   inflating: build/bin/test_mobile_nnc  
2022-05-18T03:32:41.4410231Z   inflating: build/bin/aot_model_compiler_test  
2022-05-18T03:32:41.4701633Z   inflating: build/bin/test_lazy     
2022-05-18T03:32:41.4706071Z   inflating: build/bin/torch_shm_manager  
2022-05-18T03:32:41.5384807Z   inflating: build/bin/test_tensorexpr  
2022-05-18T03:32:41.6391209Z   inflating: build/bin/test_api      
2022-05-18T03:32:41.6854943Z   inflating: build/bin/test_jit      
2022-05-18T03:32:41.6855947Z   inflating: .pytorch-test-times.json  
2022-05-18T03:32:41.6881883Z ##[group]Run df -H
2022-05-18T03:32:41.6882180Z df -H
2022-05-18T03:32:41.6897577Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-05-18T03:32:41.6897912Z env:
2022-05-18T03:32:41.6898157Z   IN_CI: 1
2022-05-18T03:32:41.6898374Z   IS_GHA: 1
2022-05-18T03:32:41.6898658Z   GIT_DEFAULT_BRANCH: master
2022-05-18T03:32:41.6898945Z ##[endgroup]
2022-05-18T03:32:41.6966044Z Filesystem      Size  Used Avail Use% Mounted on
2022-05-18T03:32:41.6966451Z devtmpfs        8.2G     0  8.2G   0% /dev
2022-05-18T03:32:41.6966809Z tmpfs           8.2G     0  8.2G   0% /dev/shm
2022-05-18T03:32:41.6967133Z tmpfs           8.2G  467k  8.2G   1% /run
2022-05-18T03:32:41.6967650Z tmpfs           8.2G     0  8.2G   0% /sys/fs/cgroup
2022-05-18T03:32:41.6968040Z /dev/nvme0n1p1  162G   14G  148G   9% /
2022-05-18T03:32:41.6968383Z tmpfs           1.7G     0  1.7G   0% /run/user/0
2022-05-18T03:32:41.6984909Z ##[group]Run .github/scripts/parse_ref.py
2022-05-18T03:32:41.6985156Z .github/scripts/parse_ref.py
2022-05-18T03:32:41.6995078Z shell: /usr/bin/bash -e {0}
2022-05-18T03:32:41.6995336Z env:
2022-05-18T03:32:41.6995494Z   IN_CI: 1
2022-05-18T03:32:41.6995655Z   IS_GHA: 1
2022-05-18T03:32:41.6995821Z   GIT_DEFAULT_BRANCH: master
2022-05-18T03:32:41.6996007Z ##[endgroup]
2022-05-18T03:32:41.7246406Z ##[group]Run set -x
2022-05-18T03:32:41.7246649Z set -x
2022-05-18T03:32:41.7246814Z 
2022-05-18T03:32:41.7247012Z if [[ $TEST_CONFIG == 'multigpu' ]]; then
2022-05-18T03:32:41.7247256Z   TEST_COMMAND=.jenkins/pytorch/multigpu-test.sh
2022-05-18T03:32:41.7247511Z elif [[ $BUILD_ENVIRONMENT == *onnx* ]]; then
2022-05-18T03:32:41.7247747Z   TEST_COMMAND=.jenkins/caffe2/test.sh
2022-05-18T03:32:41.7247933Z else
2022-05-18T03:32:41.7248134Z   TEST_COMMAND=.jenkins/pytorch/test.sh
2022-05-18T03:32:41.7248331Z fi
2022-05-18T03:32:41.7248486Z 
2022-05-18T03:32:41.7248704Z COMMIT_MESSAGES=$(git cherry -v "origin/${GIT_DEFAULT_BRANCH:-master}")
2022-05-18T03:32:41.7248970Z export COMMIT_MESSAGES
2022-05-18T03:32:41.7249148Z 
2022-05-18T03:32:41.7249359Z # detached container should get cleaned up by teardown_ec2_linux
2022-05-18T03:32:41.7249678Z # TODO: Stop building test binaries as part of the build phase
2022-05-18T03:32:41.7249952Z # Used for GPU_FLAG since that doesn't play nice
2022-05-18T03:32:41.7250180Z # shellcheck disable=SC2086,SC2090
2022-05-18T03:32:41.7250397Z container_name=$(docker run \
2022-05-18T03:32:41.7250597Z   ${GPU_FLAG:-} \
2022-05-18T03:32:41.7250778Z   -e BUILD_ENVIRONMENT \
2022-05-18T03:32:41.7250974Z   -e PR_NUMBER \
2022-05-18T03:32:41.7251186Z   -e CUSTOM_TEST_ARTIFACT_BUILD_DIR \
2022-05-18T03:32:41.7251385Z   -e GITHUB_ACTIONS \
2022-05-18T03:32:41.7251568Z   -e IN_CI \
2022-05-18T03:32:41.7251741Z   -e IS_GHA \
2022-05-18T03:32:41.7251905Z   -e BRANCH \
2022-05-18T03:32:41.7252080Z   -e SHA1 \
2022-05-18T03:32:41.7252267Z   -e AWS_DEFAULT_REGION \
2022-05-18T03:32:41.7252466Z   -e IN_WHEEL_TEST \
2022-05-18T03:32:41.7252644Z   -e SHARD_NUMBER \
2022-05-18T03:32:41.7252830Z   -e JOB_BASE_NAME \
2022-05-18T03:32:41.7253018Z   -e TEST_CONFIG \
2022-05-18T03:32:41.7253196Z   -e NUM_TEST_SHARDS \
2022-05-18T03:32:41.7253387Z   -e PR_BODY \
2022-05-18T03:32:41.7253577Z   -e COMMIT_MESSAGES \
2022-05-18T03:32:41.7253774Z   -e PYTORCH_RETRY_TEST_CASES \
2022-05-18T03:32:41.7253977Z   -e PR_LABELS \
2022-05-18T03:32:41.7254184Z   -e MAX_JOBS="$(nproc --ignore=2)" \
2022-05-18T03:32:41.7254381Z   -e SCCACHE_BUCKET \
2022-05-18T03:32:41.7254565Z   -e XLA_CUDA \
2022-05-18T03:32:41.7254769Z   -e XLA_CLANG_CACHE_S3_BUCKET_NAME \
2022-05-18T03:32:41.7255001Z   --env-file="/tmp/github_env_${GITHUB_RUN_ID}" \
2022-05-18T03:32:41.7255234Z   --ulimit stack=10485760:83886080 \
2022-05-18T03:32:41.7255461Z   --security-opt seccomp=unconfined \
2022-05-18T03:32:41.7255684Z   --cap-add=SYS_PTRACE \
2022-05-18T03:32:41.7255864Z   --ipc=host \
2022-05-18T03:32:41.7256056Z   --shm-size="${SHM_SIZE}" \
2022-05-18T03:32:41.7256242Z   --tty \
2022-05-18T03:32:41.7256402Z   --detach \
2022-05-18T03:32:41.7256598Z   --name="${container_name}" \
2022-05-18T03:32:41.7256793Z   --user jenkins \
2022-05-18T03:32:41.7257012Z   -v "${GITHUB_WORKSPACE}:/var/lib/jenkins/workspace" \
2022-05-18T03:32:41.7257262Z   -w /var/lib/jenkins/workspace \
2022-05-18T03:32:41.7257466Z   "${DOCKER_IMAGE}"
2022-05-18T03:32:41.7257626Z )
2022-05-18T03:32:41.7257872Z docker exec -t "${container_name}" sh -c "pip install dist/*.whl && ${TEST_COMMAND}"
2022-05-18T03:32:41.7268115Z shell: /usr/bin/bash -e {0}
2022-05-18T03:32:41.7268387Z env:
2022-05-18T03:32:41.7268544Z   IN_CI: 1
2022-05-18T03:32:41.7268704Z   IS_GHA: 1
2022-05-18T03:32:41.7268870Z   GIT_DEFAULT_BRANCH: master
2022-05-18T03:32:41.7269166Z   BUILD_ENVIRONMENT: linux-xenial-py3.7-gcc5.4
2022-05-18T03:32:41.7269384Z   PR_NUMBER: 
2022-05-18T03:32:41.7269553Z   BRANCH: master
2022-05-18T03:32:41.7269762Z   CUSTOM_TEST_ARTIFACT_BUILD_DIR: build/custom_test_artifacts
2022-05-18T03:32:41.7270014Z   SHA1: 3b2375291aab7b48442f2e6fb1ef66cebc761e24
2022-05-18T03:32:41.7270234Z   PYTORCH_RETRY_TEST_CASES: 1
2022-05-18T03:32:41.7270455Z   JOB_BASE_NAME: linux-xenial-py3.7-gcc5.4-test
2022-05-18T03:32:41.7270752Z   TEST_CONFIG: distributed
2022-05-18T03:32:41.7270938Z   SHARD_NUMBER: 1
2022-05-18T03:32:41.7271098Z   NUM_TEST_SHARDS: 1
2022-05-18T03:32:41.7271272Z   PR_BODY: 
2022-05-18T03:32:41.7271565Z   SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2
2022-05-18T03:32:41.7271774Z   SHM_SIZE: 1g
2022-05-18T03:32:41.7272118Z   DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-xenial-py3.7-gcc5.4:6deab82db6a72ca54cd3e3322ee4f13864536734
2022-05-18T03:32:41.7272454Z   XLA_CUDA: 
2022-05-18T03:32:41.7272719Z   XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla
2022-05-18T03:32:41.7272964Z ##[endgroup]
2022-05-18T03:32:41.7295303Z + [[ distributed == \m\u\l\t\i\g\p\u ]]
2022-05-18T03:32:41.7295930Z + [[ linux-xenial-py3.7-gcc5.4 == *onnx* ]]
2022-05-18T03:32:41.7296242Z + TEST_COMMAND=.jenkins/pytorch/test.sh
2022-05-18T03:32:41.7298330Z ++ git cherry -v origin/master
2022-05-18T03:32:41.7322894Z + COMMIT_MESSAGES=
2022-05-18T03:32:41.7323246Z + export COMMIT_MESSAGES
2022-05-18T03:32:41.7329994Z +++ nproc --ignore=2
2022-05-18T03:32:41.7360286Z ++ docker run -e BUILD_ENVIRONMENT -e PR_NUMBER -e CUSTOM_TEST_ARTIFACT_BUILD_DIR -e GITHUB_ACTIONS -e IN_CI -e IS_GHA -e BRANCH -e SHA1 -e AWS_DEFAULT_REGION -e IN_WHEEL_TEST -e SHARD_NUMBER -e JOB_BASE_NAME -e TEST_CONFIG -e NUM_TEST_SHARDS -e PR_BODY -e COMMIT_MESSAGES -e PYTORCH_RETRY_TEST_CASES -e PR_LABELS -e MAX_JOBS=6 -e SCCACHE_BUCKET -e XLA_CUDA -e XLA_CLANG_CACHE_S3_BUCKET_NAME --env-file=/tmp/github_env_2342799944 --ulimit stack=10485760:83886080 --security-opt seccomp=unconfined --cap-add=SYS_PTRACE --ipc=host --shm-size=1g --tty --detach --name= --user jenkins -v /home/ec2-user/actions-runner/_work/pytorch/pytorch:/var/lib/jenkins/workspace -w /var/lib/jenkins/workspace 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-xenial-py3.7-gcc5.4:6deab82db6a72ca54cd3e3322ee4f13864536734
2022-05-18T03:32:52.4870401Z + container_name=e67028644d5a6a7816d5dabbfe01034cc7b659531e936f1e41c2e206a19b4065
2022-05-18T03:32:52.4871156Z + docker exec -t e67028644d5a6a7816d5dabbfe01034cc7b659531e936f1e41c2e206a19b4065 sh -c 'pip install dist/*.whl && .jenkins/pytorch/test.sh'
2022-05-18T03:32:52.8559818Z Processing ./dist/torch-1.12.0a0+git3b23752-cp37-cp37m-linux_x86_64.whl
2022-05-18T03:32:52.9317981Z Requirement already satisfied: typing-extensions in /opt/conda/lib/python3.7/site-packages (from torch==1.12.0a0+git3b23752) (4.1.1)
2022-05-18T03:32:53.3015644Z Installing collected packages: torch
2022-05-18T03:32:58.7863894Z Successfully installed torch-1.12.0a0+git3b23752
2022-05-18T03:32:58.8335727Z + COMPACT_JOB_NAME=linux-xenial-py3.7-gcc5.4
2022-05-18T03:32:58.8337935Z ++ python -c 'import site; print(site.getsitepackages()[0])'
2022-05-18T03:32:58.8494969Z + TORCH_INSTALL_DIR=/opt/conda/lib/python3.7/site-packages/torch
2022-05-18T03:32:58.8495730Z + TORCH_BIN_DIR=/opt/conda/lib/python3.7/site-packages/torch/bin
2022-05-18T03:32:58.8496416Z + TORCH_LIB_DIR=/opt/conda/lib/python3.7/site-packages/torch/lib
2022-05-18T03:32:58.8497019Z + TORCH_TEST_DIR=/opt/conda/lib/python3.7/site-packages/torch/test
2022-05-18T03:32:58.8497547Z + BUILD_DIR=build
2022-05-18T03:32:58.8497928Z + BUILD_RENAMED_DIR=build_renamed
2022-05-18T03:32:58.8498253Z + BUILD_BIN_DIR=build/bin
2022-05-18T03:32:58.8498735Z + [[ -n distributed ]]
2022-05-18T03:32:58.8499300Z + BUILD_ENVIRONMENT=linux-xenial-py3.7-gcc5.4-distributed
2022-05-18T03:32:58.8500190Z + [[ linux-xenial-py3.7-gcc5.4-distributed != *bazel* ]]
2022-05-18T03:32:58.8500723Z ++ realpath build/custom_test_artifacts
2022-05-18T03:32:58.8505295Z + CUSTOM_TEST_ARTIFACT_BUILD_DIR=/var/lib/jenkins/workspace/build/custom_test_artifacts
2022-05-18T03:32:58.8508707Z ++ dirname .jenkins/pytorch/test.sh
2022-05-18T03:32:58.8513169Z + source .jenkins/pytorch/common.sh
2022-05-18T03:32:58.8516278Z +++ dirname .jenkins/pytorch/common.sh
2022-05-18T03:32:58.8523975Z ++ source .jenkins/pytorch/common_utils.sh
2022-05-18T03:32:58.8527502Z +++ TORCHVISION_COMMIT=8a2dc6f22ac4389ccba8859aa1e1cb14f1ee53db
2022-05-18T03:32:58.8528188Z ++ set -ex
2022-05-18T03:32:58.8533463Z ++++ dirname .jenkins/pytorch/common.sh
2022-05-18T03:32:58.8540840Z +++ cd .jenkins/pytorch
2022-05-18T03:32:58.8541478Z +++ pwd -P
2022-05-18T03:32:58.8543942Z ++ SCRIPT_DIR=/var/lib/jenkins/workspace/.jenkins/pytorch
2022-05-18T03:32:58.8544490Z ++ [[ linux-xenial-py3.7-gcc5.4-distributed == *linux* ]]
2022-05-18T03:32:58.8546388Z +++ find /etc/apt/ -type f -name '*.list'
2022-05-18T03:32:58.8560015Z ++ sudo sed -i 's/.*nvidia.*/# &/' /etc/apt/sources.list /etc/apt/sources.list.d/nodesource.list /etc/apt/sources.list.d/ubuntu-toolchain-r-ubuntu-test-xenial.list /etc/apt/sources.list.d/yarn.list
2022-05-18T03:32:58.8603920Z ++ [[ linux-xenial-py3.7-gcc5.4-distributed == *rocm* ]]
2022-05-18T03:32:58.8604376Z ++ echo ENTERED_USER_LAND
2022-05-18T03:32:58.8604714Z ENTERED_USER_LAND
2022-05-18T03:32:58.8605013Z ++ export IN_CI=1
2022-05-18T03:32:58.8605266Z ++ IN_CI=1
2022-05-18T03:32:58.8605932Z ++ declare -f -t trap_add
2022-05-18T03:32:58.8606463Z ++ trap_add cleanup EXIT
2022-05-18T03:32:58.8606677Z ++ trap_add_cmd=cleanup
2022-05-18T03:32:58.8606924Z ++ shift
2022-05-18T03:32:58.8607184Z ++ for trap_add_name in '"$@"'
2022-05-18T03:32:58.8611538Z ++++ trap -p EXIT
2022-05-18T03:32:58.8614716Z +++ eval 'extract_trap_cmd '
2022-05-18T03:32:58.8615043Z ++++ extract_trap_cmd
2022-05-18T03:32:58.8615314Z ++++ printf '%s\n' ''
2022-05-18T03:32:58.8615542Z +++ printf '%s\n' cleanup
2022-05-18T03:32:58.8617049Z ++ trap -- '
2022-05-18T03:32:58.8617358Z cleanup' EXIT
2022-05-18T03:32:58.8619332Z ++ [[ linux-xenial-py3.7-gcc5.4-distributed != *win-* ]]
2022-05-18T03:32:58.8619581Z ++ which sccache
2022-05-18T03:32:58.8628221Z ++ sccache --stop-server
2022-05-18T03:32:58.8647545Z ++ true
2022-05-18T03:32:58.8647897Z ++ rm -f /var/lib/jenkins/sccache_error.log
2022-05-18T03:32:58.8654184Z ++ [[ -n '' ]]
2022-05-18T03:32:58.8654788Z ++ [[ linux-xenial-py3.7-gcc5.4-distributed == *rocm* ]]
2022-05-18T03:32:58.8655156Z ++ SCCACHE_ERROR_LOG=/var/lib/jenkins/sccache_error.log
2022-05-18T03:32:58.8655433Z ++ SCCACHE_IDLE_TIMEOUT=1200
2022-05-18T03:32:58.8702464Z ++ RUST_LOG=sccache::server=error
2022-05-18T03:32:58.8702758Z ++ sccache --start-server
2022-05-18T03:32:58.8703572Z sccache: Starting the server...
2022-05-18T03:32:58.8805205Z ++ sccache --zero-stats
2022-05-18T03:32:58.8823233Z Compile requests                      0
2022-05-18T03:32:58.8823588Z Compile requests executed             0
2022-05-18T03:32:58.8823901Z Cache hits                            0
2022-05-18T03:32:58.8824177Z Cache misses                          0
2022-05-18T03:32:58.8824374Z Cache timeouts                        0
2022-05-18T03:32:58.8824561Z Cache read errors                     0
2022-05-18T03:32:58.8824757Z Forced recaches                       0
2022-05-18T03:32:58.8824953Z Cache write errors                    0
2022-05-18T03:32:58.8825200Z Compilation failures                  0
2022-05-18T03:32:58.8825542Z Cache errors                          0
2022-05-18T03:32:58.8825981Z Non-cacheable compilations            0
2022-05-18T03:32:58.8826375Z Non-cacheable calls                   0
2022-05-18T03:32:58.8826759Z Non-compilation calls                 0
2022-05-18T03:32:58.8827138Z Unsupported compiler calls            0
2022-05-18T03:32:58.8827354Z Average cache write               0.000 s
2022-05-18T03:32:58.8827703Z Average cache read miss           0.000 s
2022-05-18T03:32:58.8827902Z Average cache read hit            0.000 s
2022-05-18T03:32:58.8828114Z Failed distributed compilations       0
2022-05-18T03:32:58.8828845Z Cache location                  S3, bucket: Bucket(name=ossci-compiler-cache-circleci-v2, base_url=http://ossci-compiler-cache-circleci-v2.s3.amazonaws.com/)
2022-05-18T03:32:58.8829639Z ++ [[ linux-xenial-py3.7-gcc5.4-test == *-build ]]
2022-05-18T03:32:58.8829982Z ++ which ccache
2022-05-18T03:32:58.8836797Z ++ '[' -z linux-xenial-py3.7-gcc5.4 ']'
2022-05-18T03:32:58.8837287Z ++ [[ linux-xenial-py3.7-gcc5.4-distributed == *linux-trusty-py3.6-gcc7* ]]
2022-05-18T03:32:58.8837695Z ++ BUILD_TEST_LIBTORCH=0
2022-05-18T03:32:58.8837986Z ++ [[ distributed == *xla* ]]
2022-05-18T03:32:58.8838461Z ++ [[ linux-xenial-py3.7-gcc5.4-distributed == *centos* ]]
2022-05-18T03:32:58.8838907Z ++ [[ linux-xenial-py3.7-gcc5.4-distributed == *linux-bionic* ]]
2022-05-18T03:32:58.8839340Z ++ [[ linux-xenial-py3.7-gcc5.4-distributed == *linux-focal* ]]
2022-05-18T03:32:58.8839615Z + echo 'Testing pytorch'
2022-05-18T03:32:58.8839807Z Testing pytorch
2022-05-18T03:32:58.8840004Z + export LANG=C.UTF-8
2022-05-18T03:32:58.8840240Z + LANG=C.UTF-8
2022-05-18T03:32:58.8840923Z + PR_NUMBER=
2022-05-18T03:32:58.8841176Z + [[ distributed == \d\e\f\a\u\l\t ]]
2022-05-18T03:32:58.8841546Z + [[ distributed == \d\i\s\t\r\i\b\u\t\e\d ]]
2022-05-18T03:32:58.8841908Z + [[ linux-xenial-py3.7-gcc5.4-distributed == *rocm* ]]
2022-05-18T03:32:58.8842329Z + [[ linux-xenial-py3.7-gcc5.4-distributed == *-slow-* ]]
2022-05-18T03:32:58.8842633Z + [[ distributed == \s\l\o\w ]]
2022-05-18T03:32:58.8843043Z + [[ linux-xenial-py3.7-gcc5.4-distributed == *slow-gradcheck* ]]
2022-05-18T03:32:58.8843544Z + [[ linux-xenial-py3.7-gcc5.4-distributed == *cuda* ]]
2022-05-18T03:32:58.8844053Z + [[ linux-xenial-py3.7-gcc5.4-distributed == *rocm* ]]
2022-05-18T03:32:58.8844583Z + [[ linux-xenial-py3.7-gcc5.4-distributed == *cuda11* ]]
2022-05-18T03:32:58.8845126Z + [[ linux-xenial-py3.7-gcc5.4-distributed == *crossref* ]]
2022-05-18T03:32:58.8845466Z + [[ -n '' ]]
2022-05-18T03:32:58.8845685Z + export PYTORCH_TEST_SKIP_CUDA_MEM_LEAK_CHECK=0
2022-05-18T03:32:58.8845914Z + PYTORCH_TEST_SKIP_CUDA_MEM_LEAK_CHECK=0
2022-05-18T03:32:58.8846267Z + [[ linux-xenial-py3.7-gcc5.4-distributed == *rocm* ]]
2022-05-18T03:32:58.8846615Z + [[ linux-xenial-py3.7-gcc5.4-distributed != *ppc64le* ]]
2022-05-18T03:32:58.8846946Z + [[ linux-xenial-py3.7-gcc5.4-distributed != *-bazel-* ]]
2022-05-18T03:32:58.8847223Z + pip_install --user ninja
2022-05-18T03:32:58.8847632Z + pip install --progress-bar off --user ninja
2022-05-18T03:32:59.2524473Z Collecting ninja
2022-05-18T03:32:59.2682495Z   Downloading ninja-1.10.2.3-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl (108 kB)
2022-05-18T03:32:59.2739023Z [?25l
2022-05-18T03:32:59.5827302Z [?25hInstalling collected packages: ninja
2022-05-18T03:32:59.5909741Z   WARNING: The script ninja is installed in '/var/lib/jenkins/.local/bin' which is not on PATH.
2022-05-18T03:32:59.5910264Z   Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
2022-05-18T03:32:59.5957181Z Successfully installed ninja-1.10.2.3
2022-05-18T03:32:59.6451533Z + export PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2022-05-18T03:32:59.6452266Z + PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2022-05-18T03:32:59.6453083Z + [[ linux-xenial-py3.7-gcc5.4-distributed == *asan* ]]
2022-05-18T03:32:59.6453689Z + [[ linux-xenial-py3.7-gcc5.4-distributed == *-NO_AVX-* ]]
2022-05-18T03:32:59.6454200Z + [[ distributed == \n\o\g\p\u\_\N\O\_\A\V\X ]]
2022-05-18T03:32:59.6454577Z + [[ linux-xenial-py3.7-gcc5.4-distributed == *-NO_AVX2-* ]]
2022-05-18T03:32:59.6454836Z + [[ distributed == \n\o\g\p\u\_\N\O\_\A\V\X\2 ]]
2022-05-18T03:32:59.6455170Z + [[ linux-xenial-py3.7-gcc5.4-distributed == *-NO_AVX512-* ]]
2022-05-18T03:32:59.6455653Z + [[ distributed == \n\o\g\p\u\_\N\O\_\A\V\X\5\1\2 ]]
2022-05-18T03:32:59.6457860Z + [[ linux-xenial-py3.7-gcc5.4-distributed == *tbb* ]]
2022-05-18T03:32:59.6468999Z + [[ linux-xenial-py3.7-gcc5.4-distributed == *libtorch* ]]
2022-05-18T03:32:59.6469414Z + [[ linux-xenial-py3.7-gcc5.4-distributed == *-bazel-* ]]
2022-05-18T03:32:59.6470990Z + cd test
2022-05-18T03:32:59.6472300Z + python -c 'import torch; print(torch.__config__.show())'
2022-05-18T03:33:00.1651878Z PyTorch built with:
2022-05-18T03:33:00.1652300Z   - GCC 5.4
2022-05-18T03:33:00.1652550Z   - C++ Version: 201402
2022-05-18T03:33:00.1653001Z   - Intel(R) oneAPI Math Kernel Library Version 2022.0-Product Build 20211112 for Intel(R) 64 architecture applications
2022-05-18T03:33:00.1653718Z   - Intel(R) MKL-DNN v2.6.0 (Git Hash 52b5f107dd9cf10910aaa19cb47f3abf9b349815)
2022-05-18T03:33:00.1654059Z   - OpenMP 201307 (a.k.a. OpenMP 4.0)
2022-05-18T03:33:00.1654365Z   - LAPACK is enabled (usually provided by MKL)
2022-05-18T03:33:00.1654602Z   - NNPACK is enabled
2022-05-18T03:33:00.1654883Z   - CPU capability usage: AVX2
2022-05-18T03:33:00.1657261Z   - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CXX_COMPILER=/opt/cache/bin/c++, CXX_FLAGS= -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOCUPTI -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -DEDGE_PROFILER_USE_KINETO -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-unused-local-typedefs -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-attributes -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -Werror -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=1.12.0, USE_CUDA=OFF, USE_CUDNN=OFF, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=OFF, USE_MPI=OFF, USE_NCCL=OFF, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, 
2022-05-18T03:33:00.1658947Z 
2022-05-18T03:33:00.2478001Z + cd test
2022-05-18T03:33:00.2478417Z + python -c 'import torch; print(torch.__config__.parallel_info())'
2022-05-18T03:33:00.7448281Z ATen/Parallel:
2022-05-18T03:33:00.7448655Z 	at::get_num_threads() : 4
2022-05-18T03:33:00.7448967Z 	at::get_num_interop_threads() : 4
2022-05-18T03:33:00.7449350Z OpenMP 201307 (a.k.a. OpenMP 4.0)
2022-05-18T03:33:00.7449723Z 	omp_get_max_threads() : 4
2022-05-18T03:33:00.7450345Z Intel(R) oneAPI Math Kernel Library Version 2022.0-Product Build 20211112 for Intel(R) 64 architecture applications
2022-05-18T03:33:00.7450636Z 	mkl_get_max_threads() : 4
2022-05-18T03:33:00.7450984Z Intel(R) MKL-DNN v2.6.0 (Git Hash 52b5f107dd9cf10910aaa19cb47f3abf9b349815)
2022-05-18T03:33:00.7451466Z std::thread::hardware_concurrency() : 8
2022-05-18T03:33:00.7451862Z Environment variables:
2022-05-18T03:33:00.7452207Z 	OMP_NUM_THREADS : [not set]
2022-05-18T03:33:00.7452390Z 	MKL_NUM_THREADS : [not set]
2022-05-18T03:33:00.7452594Z ATen parallel backend: OpenMP
2022-05-18T03:33:00.7452720Z 
2022-05-18T03:33:00.8287285Z + [[ linux-xenial-py3.7-gcc5.4-distributed == *deploy* ]]
2022-05-18T03:33:00.8287911Z + [[ linux-xenial-py3.7-gcc5.4-distributed == *backward* ]]
2022-05-18T03:33:00.8288293Z + [[ distributed == *xla* ]]
2022-05-18T03:33:00.8288685Z + [[ linux-xenial-py3.7-gcc5.4-distributed == *jit_legacy-test ]]
2022-05-18T03:33:00.8289043Z + [[ linux-xenial-py3.7-gcc5.4-test == *jit_legacy-test ]]
2022-05-18T03:33:00.8289294Z + [[ distributed == \j\i\t\_\l\e\g\a\c\y ]]
2022-05-18T03:33:00.8289591Z + [[ linux-xenial-py3.7-gcc5.4-distributed == *libtorch* ]]
2022-05-18T03:33:00.8289948Z + [[ linux-xenial-py3.7-gcc5.4-distributed == *distributed* ]]
2022-05-18T03:33:00.8290383Z + test_distributed
2022-05-18T03:33:00.8290628Z + echo 'Testing distributed python tests'
2022-05-18T03:33:00.8290839Z Testing distributed python tests
2022-05-18T03:33:00.8291223Z + python test/run_test.py --distributed-tests --shard 1 1 --verbose
2022-05-18T03:33:02.8668574Z Ignoring disabled issues:  []
2022-05-18T03:33:02.8767045Z Selected tests:
2022-05-18T03:33:02.8767495Z  distributed/_shard/checkpoint/test_checkpoint
2022-05-18T03:33:02.8767895Z  distributed/_shard/checkpoint/test_file_system_checkpoint
2022-05-18T03:33:02.8768314Z  distributed/_shard/sharded_optim/test_sharded_optim
2022-05-18T03:33:02.8768571Z  distributed/_shard/sharded_tensor/ops/test_binary_cmp
2022-05-18T03:33:02.8768828Z  distributed/_shard/sharded_tensor/ops/test_chunk
2022-05-18T03:33:02.8769140Z  distributed/_shard/sharded_tensor/ops/test_elementwise_ops
2022-05-18T03:33:02.8769639Z  distributed/_shard/sharded_tensor/ops/test_embedding
2022-05-18T03:33:02.8770159Z  distributed/_shard/sharded_tensor/ops/test_embedding_bag
2022-05-18T03:33:02.8770608Z  distributed/_shard/sharded_tensor/ops/test_init
2022-05-18T03:33:02.8770934Z  distributed/_shard/sharded_tensor/ops/test_linear
2022-05-18T03:33:02.8771359Z  distributed/_shard/sharded_tensor/ops/test_math_ops
2022-05-18T03:33:02.8771707Z  distributed/_shard/sharded_tensor/ops/test_matrix_ops
2022-05-18T03:33:02.8772111Z  distributed/_shard/sharded_tensor/ops/test_softmax
2022-05-18T03:33:02.8772525Z  distributed/_shard/sharded_tensor/ops/test_tensor_ops
2022-05-18T03:33:02.8772949Z  distributed/_shard/sharded_tensor/test_megatron_prototype
2022-05-18T03:33:02.8773238Z  distributed/_shard/sharded_tensor/test_sharded_tensor
2022-05-18T03:33:02.8773493Z  distributed/_shard/sharded_tensor/test_sharded_tensor_reshard
2022-05-18T03:33:02.8773757Z  distributed/_shard/sharding_plan/test_sharding_plan
2022-05-18T03:33:02.8774014Z  distributed/_shard/sharding_spec/test_sharding_spec
2022-05-18T03:33:02.8774242Z  distributed/_shard/test_partial_tensor
2022-05-18T03:33:02.8774482Z  distributed/_shard/test_replicated_tensor
2022-05-18T03:33:02.8774706Z  distributed/_shard/test_sharder
2022-05-18T03:33:02.8774918Z  distributed/algorithms/test_join
2022-05-18T03:33:02.8775128Z  distributed/elastic/events/lib_test
2022-05-18T03:33:02.8775349Z  distributed/elastic/metrics/api_test
2022-05-18T03:33:02.8775585Z  distributed/elastic/multiprocessing/api_test
2022-05-18T03:33:02.8775805Z  distributed/elastic/timer/api_test
2022-05-18T03:33:02.8776033Z  distributed/elastic/timer/local_timer_example
2022-05-18T03:33:02.8776271Z  distributed/elastic/timer/local_timer_test
2022-05-18T03:33:02.8776493Z  distributed/elastic/utils/distributed_test
2022-05-18T03:33:02.8776721Z  distributed/elastic/utils/logging_test
2022-05-18T03:33:02.8776939Z  distributed/elastic/utils/util_test
2022-05-18T03:33:02.8777161Z  distributed/fsdp/test_distributed_checkpoint
2022-05-18T03:33:02.8777404Z  distributed/fsdp/test_flatten_params_wrapper
2022-05-18T03:33:02.8777632Z  distributed/fsdp/test_fsdp_apply
2022-05-18T03:33:02.8777838Z  distributed/fsdp/test_fsdp_checkpoint
2022-05-18T03:33:02.8778066Z  distributed/fsdp/test_fsdp_clip_grad_norm
2022-05-18T03:33:02.8778317Z  distributed/fsdp/test_fsdp_comm
2022-05-18T03:33:02.8778526Z  distributed/fsdp/test_fsdp_core
2022-05-18T03:33:02.8778728Z  distributed/fsdp/test_fsdp_exec_order
2022-05-18T03:33:02.8778965Z  distributed/fsdp/test_fsdp_freezing_weights
2022-05-18T03:33:02.8779193Z  distributed/fsdp/test_fsdp_grad_acc
2022-05-18T03:33:02.8779409Z  distributed/fsdp/test_fsdp_ignored_modules
2022-05-18T03:33:02.8779632Z  distributed/fsdp/test_fsdp_input
2022-05-18T03:33:02.8779844Z  distributed/fsdp/test_fsdp_memory
2022-05-18T03:33:02.8780045Z  distributed/fsdp/test_fsdp_meta
2022-05-18T03:33:02.8780254Z  distributed/fsdp/test_fsdp_misc
2022-05-18T03:33:02.8780480Z  distributed/fsdp/test_fsdp_mixed_precision
2022-05-18T03:33:02.8780707Z  distributed/fsdp/test_fsdp_multiple_forward
2022-05-18T03:33:02.8780953Z  distributed/fsdp/test_fsdp_multiple_wrapping
2022-05-18T03:33:02.8781353Z  distributed/fsdp/test_fsdp_optim_state
2022-05-18T03:33:02.8781575Z  distributed/fsdp/test_fsdp_overlap
2022-05-18T03:33:02.8781848Z  distributed/fsdp/test_fsdp_pure_fp16
2022-05-18T03:33:02.8782088Z  distributed/fsdp/test_fsdp_sharded_grad_scaler
2022-05-18T03:33:02.8782323Z  distributed/fsdp/test_fsdp_state_dict
2022-05-18T03:33:02.8782550Z  distributed/fsdp/test_fsdp_summon_full_params
2022-05-18T03:33:02.8782778Z  distributed/fsdp/test_fsdp_traversal
2022-05-18T03:33:02.8783264Z  distributed/fsdp/test_fsdp_uneven
2022-05-18T03:33:02.8783559Z  distributed/fsdp/test_shard_utils
2022-05-18T03:33:02.8783822Z  distributed/fsdp/test_utils
2022-05-18T03:33:02.8784166Z  distributed/fsdp/test_wrap
2022-05-18T03:33:02.8784548Z  distributed/nn/jit/test_instantiator
2022-05-18T03:33:02.8784883Z  distributed/optim/test_zero_redundancy_optimizer
2022-05-18T03:33:02.8785126Z  distributed/pipeline/sync/skip/test_api
2022-05-18T03:33:02.8785396Z  distributed/pipeline/sync/skip/test_gpipe
2022-05-18T03:33:02.8785874Z  distributed/pipeline/sync/skip/test_inspect_skip_layout
2022-05-18T03:33:02.8786301Z  distributed/pipeline/sync/skip/test_leak
2022-05-18T03:33:02.8786668Z  distributed/pipeline/sync/skip/test_portal
2022-05-18T03:33:02.8787126Z  distributed/pipeline/sync/skip/test_stash_pop
2022-05-18T03:33:02.8787521Z  distributed/pipeline/sync/skip/test_tracker
2022-05-18T03:33:02.8787777Z  distributed/pipeline/sync/skip/test_verify_skippables
2022-05-18T03:33:02.8788008Z  distributed/pipeline/sync/test_balance
2022-05-18T03:33:02.8788230Z  distributed/pipeline/sync/test_bugs
2022-05-18T03:33:02.8788461Z  distributed/pipeline/sync/test_checkpoint
2022-05-18T03:33:02.8788679Z  distributed/pipeline/sync/test_copy
2022-05-18T03:33:02.8788943Z  distributed/pipeline/sync/test_deferred_batch_norm
2022-05-18T03:33:02.8789189Z  distributed/pipeline/sync/test_dependency
2022-05-18T03:33:02.8789408Z  distributed/pipeline/sync/test_inplace
2022-05-18T03:33:02.8789641Z  distributed/pipeline/sync/test_microbatch
2022-05-18T03:33:02.8789875Z  distributed/pipeline/sync/test_phony
2022-05-18T03:33:02.8790084Z  distributed/pipeline/sync/test_pipe
2022-05-18T03:33:02.8790311Z  distributed/pipeline/sync/test_pipeline
2022-05-18T03:33:02.8790607Z  distributed/pipeline/sync/test_stream
2022-05-18T03:33:02.8790846Z  distributed/pipeline/sync/test_transparency
2022-05-18T03:33:02.8791067Z  distributed/pipeline/sync/test_worker
2022-05-18T03:33:02.8791303Z  distributed/rpc/cuda/test_tensorpipe_agent
2022-05-18T03:33:02.8791530Z  distributed/rpc/test_faulty_agent
2022-05-18T03:33:02.8791737Z  distributed/rpc/test_tensorpipe_agent
2022-05-18T03:33:02.8791949Z  distributed/test_c10d_common
2022-05-18T03:33:02.8792149Z  distributed/test_c10d_gloo
2022-05-18T03:33:02.8792332Z  distributed/test_c10d_nccl
2022-05-18T03:33:02.8792538Z  distributed/test_c10d_spawn_gloo
2022-05-18T03:33:02.8792744Z  distributed/test_c10d_spawn_nccl
2022-05-18T03:33:02.8792939Z  distributed/test_data_parallel
2022-05-18T03:33:02.8793159Z  distributed/test_distributed_spawn
2022-05-18T03:33:02.8793365Z  distributed/test_launcher
2022-05-18T03:33:02.8793543Z  distributed/test_nccl
2022-05-18T03:33:02.8793735Z  distributed/test_pg_wrapper
2022-05-18T03:33:02.8793932Z  distributed/test_store
2022-05-18T03:33:02.8856835Z Prioritized test from test file changes.
2022-05-18T03:33:02.8857257Z reordering tests for PR:
2022-05-18T03:33:02.8857629Z prioritized: []
2022-05-18T03:33:02.8865000Z the rest: ['distributed/_shard/checkpoint/test_checkpoint', 'distributed/_shard/checkpoint/test_file_system_checkpoint', 'distributed/_shard/sharded_optim/test_sharded_optim', 'distributed/_shard/sharded_tensor/ops/test_binary_cmp', 'distributed/_shard/sharded_tensor/ops/test_chunk', 'distributed/_shard/sharded_tensor/ops/test_elementwise_ops', 'distributed/_shard/sharded_tensor/ops/test_embedding', 'distributed/_shard/sharded_tensor/ops/test_embedding_bag', 'distributed/_shard/sharded_tensor/ops/test_init', 'distributed/_shard/sharded_tensor/ops/test_linear', 'distributed/_shard/sharded_tensor/ops/test_math_ops', 'distributed/_shard/sharded_tensor/ops/test_matrix_ops', 'distributed/_shard/sharded_tensor/ops/test_softmax', 'distributed/_shard/sharded_tensor/ops/test_tensor_ops', 'distributed/_shard/sharded_tensor/test_megatron_prototype', 'distributed/_shard/sharded_tensor/test_sharded_tensor', 'distributed/_shard/sharded_tensor/test_sharded_tensor_reshard', 'distributed/_shard/sharding_plan/test_sharding_plan', 'distributed/_shard/sharding_spec/test_sharding_spec', 'distributed/_shard/test_partial_tensor', 'distributed/_shard/test_replicated_tensor', 'distributed/_shard/test_sharder', 'distributed/algorithms/test_join', 'distributed/elastic/events/lib_test', 'distributed/elastic/metrics/api_test', 'distributed/elastic/multiprocessing/api_test', 'distributed/elastic/timer/api_test', 'distributed/elastic/timer/local_timer_example', 'distributed/elastic/timer/local_timer_test', 'distributed/elastic/utils/distributed_test', 'distributed/elastic/utils/logging_test', 'distributed/elastic/utils/util_test', 'distributed/fsdp/test_distributed_checkpoint', 'distributed/fsdp/test_flatten_params_wrapper', 'distributed/fsdp/test_fsdp_apply', 'distributed/fsdp/test_fsdp_checkpoint', 'distributed/fsdp/test_fsdp_clip_grad_norm', 'distributed/fsdp/test_fsdp_comm', 'distributed/fsdp/test_fsdp_core', 'distributed/fsdp/test_fsdp_exec_order', 'distributed/fsdp/test_fsdp_freezing_weights', 'distributed/fsdp/test_fsdp_grad_acc', 'distributed/fsdp/test_fsdp_ignored_modules', 'distributed/fsdp/test_fsdp_input', 'distributed/fsdp/test_fsdp_memory', 'distributed/fsdp/test_fsdp_meta', 'distributed/fsdp/test_fsdp_misc', 'distributed/fsdp/test_fsdp_mixed_precision', 'distributed/fsdp/test_fsdp_multiple_forward', 'distributed/fsdp/test_fsdp_multiple_wrapping', 'distributed/fsdp/test_fsdp_optim_state', 'distributed/fsdp/test_fsdp_overlap', 'distributed/fsdp/test_fsdp_pure_fp16', 'distributed/fsdp/test_fsdp_sharded_grad_scaler', 'distributed/fsdp/test_fsdp_state_dict', 'distributed/fsdp/test_fsdp_summon_full_params', 'distributed/fsdp/test_fsdp_traversal', 'distributed/fsdp/test_fsdp_uneven', 'distributed/fsdp/test_shard_utils', 'distributed/fsdp/test_utils', 'distributed/fsdp/test_wrap', 'distributed/nn/jit/test_instantiator', 'distributed/optim/test_zero_redundancy_optimizer', 'distributed/pipeline/sync/skip/test_api', 'distributed/pipeline/sync/skip/test_gpipe', 'distributed/pipeline/sync/skip/test_inspect_skip_layout', 'distributed/pipeline/sync/skip/test_leak', 'distributed/pipeline/sync/skip/test_portal', 'distributed/pipeline/sync/skip/test_stash_pop', 'distributed/pipeline/sync/skip/test_tracker', 'distributed/pipeline/sync/skip/test_verify_skippables', 'distributed/pipeline/sync/test_balance', 'distributed/pipeline/sync/test_bugs', 'distributed/pipeline/sync/test_checkpoint', 'distributed/pipeline/sync/test_copy', 'distributed/pipeline/sync/test_deferred_batch_norm', 'distributed/pipeline/sync/test_dependency', 'distributed/pipeline/sync/test_inplace', 'distributed/pipeline/sync/test_microbatch', 'distributed/pipeline/sync/test_phony', 'distributed/pipeline/sync/test_pipe', 'distributed/pipeline/sync/test_pipeline', 'distributed/pipeline/sync/test_stream', 'distributed/pipeline/sync/test_transparency', 'distributed/pipeline/sync/test_worker', 'distributed/rpc/cuda/test_tensorpipe_agent', 'distributed/rpc/test_faulty_agent', 'distributed/rpc/test_tensorpipe_agent', 'distributed/test_c10d_common', 'distributed/test_c10d_gloo', 'distributed/test_c10d_nccl', 'distributed/test_c10d_spawn_gloo', 'distributed/test_c10d_spawn_nccl', 'distributed/test_data_parallel', 'distributed/test_distributed_spawn', 'distributed/test_launcher', 'distributed/test_nccl', 'distributed/test_pg_wrapper', 'distributed/test_store']
2022-05-18T03:33:02.8869481Z 
2022-05-18T03:33:02.9285967Z Running distributed/_shard/checkpoint/test_checkpoint ... [2022-05-18 03:33:02.928320]
2022-05-18T03:33:02.9286526Z Executing ['/opt/conda/bin/python', 'distributed/_shard/checkpoint/test_checkpoint.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-05-18 03:33:02.928368]
2022-05-18T03:33:03.5124737Z Test results will be stored in test-reports/python-unittest/distributed._shard.checkpoint.test_checkpoint
2022-05-18T03:33:03.5136878Z 
2022-05-18T03:33:03.5137009Z Running tests...
2022-05-18T03:33:03.5137394Z ----------------------------------------------------------------------
2022-05-18T03:33:03.5144743Z   test_checkpoint_has_shard_overlap (__main__.TestDistributedCheckpointing) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-05-18T03:33:03.5151087Z   test_checkpoint_has_shard_too_small (__main__.TestDistributedCheckpointing) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-05-18T03:33:03.5157739Z   test_checkpoint_has_storage_type_mismatch (__main__.TestDistributedCheckpointing) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-05-18T03:33:03.5172170Z   test_storage_key_mapping (__main__.TestDistributedCheckpointing) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-05-18T03:33:03.5179496Z   test_tensor_metadata_with_missing_rank_spec (__main__.TestDistributedCheckpointing) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-05-18T03:33:03.5191280Z   test_validate_metadata (__main__.TestDistributedCheckpointing) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-05-18T03:33:03.7947325Z   test_create_key_handles_collision (__main__.TestStorageKeys) ... ok (0.275s)
2022-05-18T03:33:03.7947582Z 
2022-05-18T03:33:03.7947945Z ----------------------------------------------------------------------
2022-05-18T03:33:03.7948209Z Ran 7 tests in 0.281s
2022-05-18T03:33:03.7948324Z 
2022-05-18T03:33:03.7948404Z OK (skipped=6)
2022-05-18T03:33:03.7948543Z 
2022-05-18T03:33:03.7948639Z Generating XML reports...
2022-05-18T03:33:03.7977746Z Generated XML report: test-reports/python-unittest/distributed._shard.checkpoint.test_checkpoint/TEST-TestStorageKeys-20220518033303.xml
2022-05-18T03:33:03.7986437Z Generated XML report: test-reports/python-unittest/distributed._shard.checkpoint.test_checkpoint/TEST-TestDistributedCheckpointing-20220518033303.xml
2022-05-18T03:33:03.9728999Z Running distributed/_shard/checkpoint/test_file_system_checkpoint ... [2022-05-18 03:33:03.972535]
2022-05-18T03:33:03.9729613Z Executing ['/opt/conda/bin/python', 'distributed/_shard/checkpoint/test_file_system_checkpoint.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-05-18 03:33:03.972611]
2022-05-18T03:33:04.5459993Z Test results will be stored in test-reports/python-unittest/distributed._shard.checkpoint.test_file_system_checkpoint
2022-05-18T03:33:04.5473539Z 
2022-05-18T03:33:04.5473633Z Running tests...
2022-05-18T03:33:04.5474626Z ----------------------------------------------------------------------
2022-05-18T03:33:04.5487782Z   test_load_rowwise_to_colwise (__main__.TestDistributedReshardOnLoad) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-05-18T03:33:04.5510063Z   test_load_with_different_shard_plan (__main__.TestDistributedReshardOnLoad) ... skip: c10d was not compiled with the NCCL backend (0.002s)
2022-05-18T03:33:04.5516966Z   test_save_load_bytes (__main__.TestDistributedReshardOnLoad) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-05-18T03:33:04.8431906Z   test_read_write_only_tensor (__main__.TestDistributedStateDictSaveLoad) ... ok (0.291s)
2022-05-18T03:33:04.8443694Z   test_read_write_shard_tensor (__main__.TestDistributedStateDictSaveLoadWithSharedTensor) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-05-18T03:33:04.8444221Z 
2022-05-18T03:33:04.8444698Z ----------------------------------------------------------------------
2022-05-18T03:33:04.8445136Z Ran 5 tests in 0.297s
2022-05-18T03:33:04.8445345Z 
2022-05-18T03:33:04.8445473Z OK (skipped=4)
2022-05-18T03:33:04.8445667Z 
2022-05-18T03:33:04.8445805Z Generating XML reports...
2022-05-18T03:33:04.8475917Z Generated XML report: test-reports/python-unittest/distributed._shard.checkpoint.test_file_system_checkpoint/TEST-TestDistributedStateDictSaveLoad-20220518033304.xml
2022-05-18T03:33:04.8481535Z Generated XML report: test-reports/python-unittest/distributed._shard.checkpoint.test_file_system_checkpoint/TEST-TestDistributedReshardOnLoad-20220518033304.xml
2022-05-18T03:33:04.8485126Z Generated XML report: test-reports/python-unittest/distributed._shard.checkpoint.test_file_system_checkpoint/TEST-TestDistributedStateDictSaveLoadWithSharedTensor-20220518033304.xml
2022-05-18T03:33:05.0067198Z Running distributed/_shard/sharded_optim/test_sharded_optim ... [2022-05-18 03:33:05.006271]
2022-05-18T03:33:05.0067788Z Executing ['/opt/conda/bin/python', 'distributed/_shard/sharded_optim/test_sharded_optim.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-05-18 03:33:05.006365]
2022-05-18T03:33:05.5699573Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharded_optim.test_sharded_optim
2022-05-18T03:33:05.5710604Z 
2022-05-18T03:33:05.5711018Z Running tests...
2022-05-18T03:33:05.5711474Z ----------------------------------------------------------------------
2022-05-18T03:33:05.5722385Z   test_named_params_with_sharded_tensor (__main__.TestShardedOptimizer) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-05-18T03:33:05.5738061Z   test_sharded_optim (__main__.TestShardedOptimizer) ... skip: c10d was not compiled with the NCCL backend (0.002s)
2022-05-18T03:33:05.5738301Z 
2022-05-18T03:33:05.5738639Z ----------------------------------------------------------------------
2022-05-18T03:33:05.5738875Z Ran 2 tests in 0.003s
2022-05-18T03:33:05.5738990Z 
2022-05-18T03:33:05.5739062Z OK (skipped=2)
2022-05-18T03:33:05.5739173Z 
2022-05-18T03:33:05.5739257Z Generating XML reports...
2022-05-18T03:33:05.5765032Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_optim.test_sharded_optim/TEST-TestShardedOptimizer-20220518033305.xml
2022-05-18T03:33:05.6722257Z Running distributed/_shard/sharded_tensor/ops/test_binary_cmp ... [2022-05-18 03:33:05.671868]
2022-05-18T03:33:05.6722885Z Executing ['/opt/conda/bin/python', 'distributed/_shard/sharded_tensor/ops/test_binary_cmp.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-05-18 03:33:05.671944]
2022-05-18T03:33:06.2363280Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_binary_cmp
2022-05-18T03:33:06.2373673Z 
2022-05-18T03:33:06.2373787Z Running tests...
2022-05-18T03:33:06.2374379Z ----------------------------------------------------------------------
2022-05-18T03:33:06.2383080Z   test_torch_allclose (__main__.TestShardedTensorBinaryOps)
2022-05-18T03:33:06.2384149Z Test torch.allclose(ShardedTensor, ShardedTensor) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-05-18T03:33:06.2386598Z   test_torch_allclose_tensor_specs (__main__.TestShardedTensorBinaryOps) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-05-18T03:33:06.2391033Z   test_torch_equal (__main__.TestShardedTensorBinaryOps)
2022-05-18T03:33:06.2391704Z Test torch.equal(ShardedTensor, ShardedTensor) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-05-18T03:33:06.2395164Z   test_torch_equal_tensor_specs (__main__.TestShardedTensorBinaryOps) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-05-18T03:33:06.2395622Z 
2022-05-18T03:33:06.2396032Z ----------------------------------------------------------------------
2022-05-18T03:33:06.2396446Z Ran 4 tests in 0.002s
2022-05-18T03:33:06.2396632Z 
2022-05-18T03:33:06.2396733Z OK (skipped=4)
2022-05-18T03:33:06.2396904Z 
2022-05-18T03:33:06.2397042Z Generating XML reports...
2022-05-18T03:33:06.2424370Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_binary_cmp/TEST-TestShardedTensorBinaryOps-20220518033306.xml
2022-05-18T03:33:06.3388883Z Running distributed/_shard/sharded_tensor/ops/test_chunk ... [2022-05-18 03:33:06.338520]
2022-05-18T03:33:06.3389487Z Executing ['/opt/conda/bin/python', 'distributed/_shard/sharded_tensor/ops/test_chunk.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-05-18 03:33:06.338598]
2022-05-18T03:33:06.9031495Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_chunk
2022-05-18T03:33:06.9041470Z 
2022-05-18T03:33:06.9041686Z Running tests...
2022-05-18T03:33:06.9042132Z ----------------------------------------------------------------------
2022-05-18T03:33:06.9049746Z   test_sharded_chunk (__main__.TestShardedTensorChunkOps) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-05-18T03:33:06.9056850Z   test_sharded_chunk_error (__main__.TestShardedTensorChunkOps) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-05-18T03:33:06.9057220Z 
2022-05-18T03:33:06.9057478Z ----------------------------------------------------------------------
2022-05-18T03:33:06.9057726Z Ran 2 tests in 0.001s
2022-05-18T03:33:06.9057840Z 
2022-05-18T03:33:06.9057900Z OK (skipped=2)
2022-05-18T03:33:06.9058016Z 
2022-05-18T03:33:06.9058101Z Generating XML reports...
2022-05-18T03:33:06.9083117Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_chunk/TEST-TestShardedTensorChunkOps-20220518033306.xml
2022-05-18T03:33:07.0040981Z Running distributed/_shard/sharded_tensor/ops/test_elementwise_ops ... [2022-05-18 03:33:07.003669]
2022-05-18T03:33:07.0042024Z Executing ['/opt/conda/bin/python', 'distributed/_shard/sharded_tensor/ops/test_elementwise_ops.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-05-18 03:33:07.003750]
2022-05-18T03:33:07.5721604Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_elementwise_ops
2022-05-18T03:33:07.5732156Z 
2022-05-18T03:33:07.5732246Z Running tests...
2022-05-18T03:33:07.5733558Z ----------------------------------------------------------------------
2022-05-18T03:33:07.5743155Z   test_sharded_dropout (__main__.TestShardedTensorElementWiseOps) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-05-18T03:33:07.5750641Z   test_sharded_gelu (__main__.TestShardedTensorElementWiseOps) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-05-18T03:33:07.5757344Z   test_sharded_relu (__main__.TestShardedTensorElementWiseOps) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-05-18T03:33:07.5757600Z 
2022-05-18T03:33:07.5757850Z ----------------------------------------------------------------------
2022-05-18T03:33:07.5758085Z Ran 3 tests in 0.002s
2022-05-18T03:33:07.5758199Z 
2022-05-18T03:33:07.5758273Z OK (skipped=3)
2022-05-18T03:33:07.5758381Z 
2022-05-18T03:33:07.5758467Z Generating XML reports...
2022-05-18T03:33:07.5784856Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_elementwise_ops/TEST-TestShardedTensorElementWiseOps-20220518033307.xml
2022-05-18T03:33:07.6744751Z Running distributed/_shard/sharded_tensor/ops/test_embedding ... [2022-05-18 03:33:07.674047]
2022-05-18T03:33:07.6745522Z Executing ['/opt/conda/bin/python', 'distributed/_shard/sharded_tensor/ops/test_embedding.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-05-18 03:33:07.674125]
2022-05-18T03:33:08.2412734Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_embedding
2022-05-18T03:33:08.2423981Z 
2022-05-18T03:33:08.2424142Z Running tests...
2022-05-18T03:33:08.2424751Z ----------------------------------------------------------------------
2022-05-18T03:33:08.2437064Z   test_sharded_embedding_colwise (__main__.TestShardedEmbedding) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-05-18T03:33:08.2450585Z   test_sharded_embedding_rowwise (__main__.TestShardedEmbedding) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-05-18T03:33:08.2450976Z 
2022-05-18T03:33:08.2451414Z ----------------------------------------------------------------------
2022-05-18T03:33:08.2451755Z Ran 2 tests in 0.003s
2022-05-18T03:33:08.2452036Z 
2022-05-18T03:33:08.2452109Z OK (skipped=2)
2022-05-18T03:33:08.2452216Z 
2022-05-18T03:33:08.2452301Z Generating XML reports...
2022-05-18T03:33:08.2478630Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_embedding/TEST-TestShardedEmbedding-20220518033308.xml
2022-05-18T03:33:08.3456466Z Running distributed/_shard/sharded_tensor/ops/test_embedding_bag ... [2022-05-18 03:33:08.345260]
2022-05-18T03:33:08.3457126Z Executing ['/opt/conda/bin/python', 'distributed/_shard/sharded_tensor/ops/test_embedding_bag.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-05-18 03:33:08.345336]
2022-05-18T03:33:08.9140636Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_embedding_bag
2022-05-18T03:33:08.9151486Z 
2022-05-18T03:33:08.9151827Z Running tests...
2022-05-18T03:33:08.9152263Z ----------------------------------------------------------------------
2022-05-18T03:33:08.9155942Z   test_sharded_embedding_bag_colwise (__main__.TestShardedEmbeddingBag) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-05-18T03:33:08.9159534Z   test_sharded_embedding_bag_rowwise (__main__.TestShardedEmbeddingBag) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-05-18T03:33:08.9159915Z 
2022-05-18T03:33:08.9160140Z ----------------------------------------------------------------------
2022-05-18T03:33:08.9160384Z Ran 2 tests in 0.001s
2022-05-18T03:33:08.9160502Z 
2022-05-18T03:33:08.9160576Z OK (skipped=2)
2022-05-18T03:33:08.9160683Z 
2022-05-18T03:33:08.9160768Z Generating XML reports...
2022-05-18T03:33:08.9186209Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_embedding_bag/TEST-TestShardedEmbeddingBag-20220518033308.xml
2022-05-18T03:33:09.0143670Z Running distributed/_shard/sharded_tensor/ops/test_init ... [2022-05-18 03:33:09.014008]
2022-05-18T03:33:09.0144251Z Executing ['/opt/conda/bin/python', 'distributed/_shard/sharded_tensor/ops/test_init.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-05-18 03:33:09.014087]
2022-05-18T03:33:09.5784386Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_init
2022-05-18T03:33:09.5794796Z 
2022-05-18T03:33:09.5794963Z Running tests...
2022-05-18T03:33:09.5795302Z ----------------------------------------------------------------------
2022-05-18T03:33:09.5806922Z   test_init_sharded_tensor_with_kaiming_uniform (__main__.TestShardedTensorNNInit)
2022-05-18T03:33:09.5807697Z Test torch.nn.init.kaiming_uniform_(ShardedTensor, a, mode, nonlinearit) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-05-18T03:33:09.5817043Z   test_init_sharded_tensor_with_normal (__main__.TestShardedTensorNNInit)
2022-05-18T03:33:09.5817485Z Test torch.nn.init.normal_(ShardedTensor, mean, std) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-05-18T03:33:09.5826690Z   test_init_sharded_tensor_with_uniform (__main__.TestShardedTensorNNInit)
2022-05-18T03:33:09.5827367Z Test torch.nn.init.uniform_(ShardedTensor, a, b) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-05-18T03:33:09.5827684Z 
2022-05-18T03:33:09.5827958Z ----------------------------------------------------------------------
2022-05-18T03:33:09.5828196Z Ran 3 tests in 0.003s
2022-05-18T03:33:09.5828315Z 
2022-05-18T03:33:09.5828389Z OK (skipped=3)
2022-05-18T03:33:09.5828496Z 
2022-05-18T03:33:09.5828582Z Generating XML reports...
2022-05-18T03:33:09.5855071Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_init/TEST-TestShardedTensorNNInit-20220518033309.xml
2022-05-18T03:33:09.6816749Z Running distributed/_shard/sharded_tensor/ops/test_linear ... [2022-05-18 03:33:09.681343]
2022-05-18T03:33:09.6817347Z Executing ['/opt/conda/bin/python', 'distributed/_shard/sharded_tensor/ops/test_linear.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-05-18 03:33:09.681419]
2022-05-18T03:33:10.2517020Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_linear
2022-05-18T03:33:10.2527776Z 
2022-05-18T03:33:10.2528126Z Running tests...
2022-05-18T03:33:10.2528755Z ----------------------------------------------------------------------
2022-05-18T03:33:10.2542211Z   test_sharded_linear_colwise (__main__.TestShardedTensorOpsLinear) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-05-18T03:33:10.2568902Z   test_sharded_linear_errors (__main__.TestShardedTensorOpsLinear) ... skip: c10d was not compiled with the NCCL backend (0.003s)
2022-05-18T03:33:10.2579548Z   test_sharded_linear_rowwise (__main__.TestShardedTensorOpsLinear) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-05-18T03:33:10.2579997Z 
2022-05-18T03:33:10.2580267Z ----------------------------------------------------------------------
2022-05-18T03:33:10.2580519Z Ran 3 tests in 0.005s
2022-05-18T03:33:10.2580646Z 
2022-05-18T03:33:10.2580708Z OK (skipped=3)
2022-05-18T03:33:10.2580817Z 
2022-05-18T03:33:10.2580905Z Generating XML reports...
2022-05-18T03:33:10.2606281Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_linear/TEST-TestShardedTensorOpsLinear-20220518033310.xml
2022-05-18T03:33:10.3596545Z Running distributed/_shard/sharded_tensor/ops/test_math_ops ... [2022-05-18 03:33:10.359240]
2022-05-18T03:33:10.3597126Z Executing ['/opt/conda/bin/python', 'distributed/_shard/sharded_tensor/ops/test_math_ops.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-05-18 03:33:10.359314]
2022-05-18T03:33:11.0157677Z Running distributed/_shard/sharded_tensor/ops/test_matrix_ops ... [2022-05-18 03:33:11.015338]
2022-05-18T03:33:11.0158338Z Executing ['/opt/conda/bin/python', 'distributed/_shard/sharded_tensor/ops/test_matrix_ops.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-05-18 03:33:11.015415]
2022-05-18T03:33:11.5816651Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_matrix_ops
2022-05-18T03:33:11.5829030Z 
2022-05-18T03:33:11.5829127Z Running tests...
2022-05-18T03:33:11.5829754Z ----------------------------------------------------------------------
2022-05-18T03:33:11.5836575Z   test_sharded_tensor_contiguous (__main__.TestShardedTensorMatrixOps) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-05-18T03:33:11.5847578Z   test_sharded_tensor_layer_norm (__main__.TestShardedTensorMatrixOps) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-05-18T03:33:11.5855075Z   test_sharded_tensor_layer_norm_error (__main__.TestShardedTensorMatrixOps) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-05-18T03:33:11.5859304Z   test_sharded_tensor_masked_fill (__main__.TestShardedTensorMatrixOps) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-05-18T03:33:11.5868137Z   test_sharded_tensor_masked_fill_error (__main__.TestShardedTensorMatrixOps) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-05-18T03:33:11.5874776Z   test_sharded_tensor_softmax (__main__.TestShardedTensorMatrixOps) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-05-18T03:33:11.5884630Z   test_sharded_tensor_transpose (__main__.TestShardedTensorMatrixOps) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-05-18T03:33:11.5889501Z   test_sharded_tensor_transpose_error (__main__.TestShardedTensorMatrixOps) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-05-18T03:33:11.5897503Z   test_sharded_tensor_type_as (__main__.TestShardedTensorMatrixOps) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-05-18T03:33:11.5906188Z   test_sharded_tensor_view (__main__.TestShardedTensorMatrixOps) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-05-18T03:33:11.5914335Z   test_sharded_tensor_view_error (__main__.TestShardedTensorMatrixOps) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-05-18T03:33:11.5915046Z 
2022-05-18T03:33:11.5915347Z ----------------------------------------------------------------------
2022-05-18T03:33:11.5915598Z Ran 11 tests in 0.008s
2022-05-18T03:33:11.5915788Z 
2022-05-18T03:33:11.5915869Z OK (skipped=11)
2022-05-18T03:33:11.5915976Z 
2022-05-18T03:33:11.5916049Z Generating XML reports...
2022-05-18T03:33:11.5953458Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_matrix_ops/TEST-TestShardedTensorMatrixOps-20220518033311.xml
2022-05-18T03:33:11.6903396Z Running distributed/_shard/sharded_tensor/ops/test_softmax ... [2022-05-18 03:33:11.689885]
2022-05-18T03:33:11.6904409Z Executing ['/opt/conda/bin/python', 'distributed/_shard/sharded_tensor/ops/test_softmax.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-05-18 03:33:11.689960]
2022-05-18T03:33:12.2447568Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb4ijlnbu
2022-05-18T03:33:12.2448379Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb4ijlnbu/_remote_module_non_scriptable.py
2022-05-18T03:33:12.2548732Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_softmax
2022-05-18T03:33:12.2560740Z 
2022-05-18T03:33:12.2560826Z Running tests...
2022-05-18T03:33:12.2561789Z ----------------------------------------------------------------------
2022-05-18T03:33:12.2565553Z   test_sharded_softmax_basic (__main__.TestShardedSoftmax) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-05-18T03:33:12.2569090Z   test_sharded_softmax_on_sharding_dim (__main__.TestShardedSoftmax) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-05-18T03:33:12.2569425Z 
2022-05-18T03:33:12.2569846Z ----------------------------------------------------------------------
2022-05-18T03:33:12.2570267Z Ran 2 tests in 0.001s
2022-05-18T03:33:12.2570369Z 
2022-05-18T03:33:12.2570441Z OK (skipped=2)
2022-05-18T03:33:12.2570554Z 
2022-05-18T03:33:12.2570639Z Generating XML reports...
2022-05-18T03:33:12.2595969Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_softmax/TEST-TestShardedSoftmax-20220518033312.xml
2022-05-18T03:33:12.3626712Z Running distributed/_shard/sharded_tensor/ops/test_tensor_ops ... [2022-05-18 03:33:12.362272]
2022-05-18T03:33:12.3627306Z Executing ['/opt/conda/bin/python', 'distributed/_shard/sharded_tensor/ops/test_tensor_ops.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-05-18 03:33:12.362359]
2022-05-18T03:33:12.9284658Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_tensor_ops
2022-05-18T03:33:12.9295258Z 
2022-05-18T03:33:12.9295386Z Running tests...
2022-05-18T03:33:12.9295831Z ----------------------------------------------------------------------
2022-05-18T03:33:12.9303255Z   test_clone (__main__.TestTensorOps) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-05-18T03:33:12.9309331Z   test_deep_copy (__main__.TestTensorOps) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-05-18T03:33:12.9316801Z   test_detach (__main__.TestTensorOps) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-05-18T03:33:12.9324288Z   test_set_requires_grad (__main__.TestTensorOps) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-05-18T03:33:12.9324662Z 
2022-05-18T03:33:12.9325104Z ----------------------------------------------------------------------
2022-05-18T03:33:12.9325362Z Ran 4 tests in 0.003s
2022-05-18T03:33:12.9325477Z 
2022-05-18T03:33:12.9325550Z OK (skipped=4)
2022-05-18T03:33:12.9325657Z 
2022-05-18T03:33:12.9325741Z Generating XML reports...
2022-05-18T03:33:12.9352650Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_tensor_ops/TEST-TestTensorOps-20220518033312.xml
2022-05-18T03:33:13.0308834Z Running distributed/_shard/sharded_tensor/test_megatron_prototype ... [2022-05-18 03:33:13.030537]
2022-05-18T03:33:13.0309739Z Executing ['/opt/conda/bin/python', 'distributed/_shard/sharded_tensor/test_megatron_prototype.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-05-18 03:33:13.030619]
2022-05-18T03:33:13.5976325Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharded_tensor.test_megatron_prototype
2022-05-18T03:33:13.5987444Z 
2022-05-18T03:33:13.5987682Z Running tests...
2022-05-18T03:33:13.5988313Z ----------------------------------------------------------------------
2022-05-18T03:33:13.5997061Z   test_megatron_two_layer_prototype (__main__.TestShardedTensorMegatronLinear) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-05-18T03:33:13.5997523Z 
2022-05-18T03:33:13.5997913Z ----------------------------------------------------------------------
2022-05-18T03:33:13.5998329Z Ran 1 test in 0.001s
2022-05-18T03:33:13.5998521Z 
2022-05-18T03:33:13.5998640Z OK (skipped=1)
2022-05-18T03:33:13.5998829Z 
2022-05-18T03:33:13.5998955Z Generating XML reports...
2022-05-18T03:33:13.6023599Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.test_megatron_prototype/TEST-TestShardedTensorMegatronLinear-20220518033313.xml
2022-05-18T03:33:13.6998253Z Running distributed/_shard/sharded_tensor/test_sharded_tensor ... [2022-05-18 03:33:13.699403]
2022-05-18T03:33:13.6998876Z Executing ['/opt/conda/bin/python', 'distributed/_shard/sharded_tensor/test_sharded_tensor.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-05-18 03:33:13.699478]
2022-05-18T03:33:14.2807091Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor
2022-05-18T03:33:14.2831208Z 
2022-05-18T03:33:14.2831325Z Running tests...
2022-05-18T03:33:14.2831740Z ----------------------------------------------------------------------
2022-05-18T03:33:14.2840277Z   test_empty (__main__.TestCreateTensorFromParams) ... skip: CUDA GPU is needed (0.001s)
2022-05-18T03:33:14.2846595Z   test_local_tensor (__main__.TestLocalTensor) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-05-18T03:33:14.2851971Z   test_local_tensor_error (__main__.TestLocalTensor) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-05-18T03:33:14.2858176Z   test_collect_local_shard (__main__.TestModuleHookApi) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-05-18T03:33:14.2867793Z   test_reshard_output (__main__.TestModuleHookApi) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-05-18T03:33:14.2877705Z   test_shard_parameter (__main__.TestShardParameter) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-05-18T03:33:14.2891563Z   test_shard_parameter_errors (__main__.TestShardParameter) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-05-18T03:33:14.2899667Z   test_shard_tensor (__main__.TestShardTensor) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-05-18T03:33:14.2911357Z   test_shard_tensor_errors (__main__.TestShardTensor) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-05-18T03:33:14.2917595Z   test_cleanup (__main__.TestShardedTensorChunked) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-05-18T03:33:14.2934494Z   test_complete_world_size (__main__.TestShardedTensorChunked) ... skip: c10d was not compiled with the NCCL backend (0.002s)
2022-05-18T03:33:14.2949944Z   test_create_sharded_tensor_like (__main__.TestShardedTensorChunked)
2022-05-18T03:33:14.2950395Z Test tensor like methods, i.e. torch.zeros_like(...), torch.full_like, etc. ... skip: c10d was not compiled with the NCCL backend (0.002s)
2022-05-18T03:33:14.2959395Z   test_create_sharded_tensor_with_full (__main__.TestShardedTensorChunked)
2022-05-18T03:33:14.2959736Z Test sharded_tensor.full(...) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-05-18T03:33:14.2967854Z   test_create_sharded_tensor_with_ones (__main__.TestShardedTensorChunked)
2022-05-18T03:33:14.2968421Z Test sharded_tensor.ones(...) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-05-18T03:33:14.2982194Z   test_create_sharded_tensor_with_rand (__main__.TestShardedTensorChunked)
2022-05-18T03:33:14.2982670Z Test sharded_tensor.rand(...)/randn(...) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-05-18T03:33:14.2990225Z   test_create_sharded_tensor_with_zeros (__main__.TestShardedTensorChunked)
2022-05-18T03:33:14.2990927Z Test sharded_tensor.zeros(...) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-05-18T03:33:14.2997980Z   test_gather_even (__main__.TestShardedTensorChunked)
2022-05-18T03:33:14.2998656Z Test _sharded_tensor.gather(...) with evenly distributed._shards ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-05-18T03:33:14.3005868Z   test_gather_uneven (__main__.TestShardedTensorChunked)
2022-05-18T03:33:14.3006523Z Test _sharded_tensor.gather(...) with unevenly distributed._shards ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-05-18T03:33:14.3016970Z   test_insufficient_sharding_dims (__main__.TestShardedTensorChunked) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-05-18T03:33:14.3023047Z   test_invalid_pg_rpc_ranks (__main__.TestShardedTensorChunked) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-05-18T03:33:14.3041887Z   test_invalid_sharding (__main__.TestShardedTensorChunked) ... skip: c10d was not compiled with the NCCL backend (0.002s)
2022-05-18T03:33:14.3052837Z   test_load_state_dict_errors (__main__.TestShardedTensorChunked) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-05-18T03:33:14.3066589Z   test_multiple_local_shards (__main__.TestShardedTensorChunked) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-05-18T03:33:14.3082945Z   test_new_group (__main__.TestShardedTensorChunked) ... skip: c10d was not compiled with the NCCL backend (0.002s)
2022-05-18T03:33:14.3098564Z   test_partial_world_size (__main__.TestShardedTensorChunked) ... skip: c10d was not compiled with the NCCL backend (0.002s)
2022-05-18T03:33:14.3111935Z   test_sharded_tensor_metadata (__main__.TestShardedTensorChunked) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-05-18T03:33:14.3126839Z   test_sharded_tensor_sizes (__main__.TestShardedTensorChunked) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-05-18T03:33:14.3136788Z   test_sharding_columns (__main__.TestShardedTensorChunked) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-05-18T03:33:14.3144818Z   test_state_dict (__main__.TestShardedTensorChunked) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-05-18T03:33:14.3154034Z   test_state_dict_new_group (__main__.TestShardedTensorChunked) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-05-18T03:33:14.3161775Z   test_state_dict_no_sharded_tensors (__main__.TestShardedTensorChunked) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-05-18T03:33:14.3168728Z   test_custom_op (__main__.TestShardedTensorCustomOps) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-05-18T03:33:14.3174463Z   test_custom_op_errors (__main__.TestShardedTensorCustomOps) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-05-18T03:33:14.3181432Z   test_custom_op_override (__main__.TestShardedTensorCustomOps) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-05-18T03:33:14.3192941Z   test_create_sharded_tensor_with_ones (__main__.TestShardedTensorEnumerable)
2022-05-18T03:33:14.3193311Z Test sharded_tensor.ones(...) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-05-18T03:33:14.3204021Z   test_gather_even (__main__.TestShardedTensorEnumerable)
2022-05-18T03:33:14.3204403Z Test _sharded_tensor.gather(...) with evenly distributed._shards ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-05-18T03:33:14.3215076Z   test_gather_uneven (__main__.TestShardedTensorEnumerable)
2022-05-18T03:33:14.3215657Z Test _sharded_tensor.gather(...) with unevenly distributed._shards ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-05-18T03:33:14.3234866Z   test_grid_sharding (__main__.TestShardedTensorEnumerable) ... skip: c10d was not compiled with the NCCL backend (0.002s)
2022-05-18T03:33:14.3256234Z   test_multiple_local_shards (__main__.TestShardedTensorEnumerable) ... skip: c10d was not compiled with the NCCL backend (0.002s)
2022-05-18T03:33:14.3274914Z   test_new_group (__main__.TestShardedTensorEnumerable) ... skip: c10d was not compiled with the NCCL backend (0.002s)
2022-05-18T03:33:14.3294187Z   test_partial_world_size (__main__.TestShardedTensorEnumerable) ... skip: c10d was not compiled with the NCCL backend (0.002s)
2022-05-18T03:33:14.3313290Z   test_sharded_tensor_metadata (__main__.TestShardedTensorEnumerable) ... skip: c10d was not compiled with the NCCL backend (0.002s)
2022-05-18T03:33:14.3334760Z   test_sharded_tensor_to_cpu (__main__.TestShardedTensorEnumerable) ... skip: c10d was not compiled with the NCCL backend (0.002s)
2022-05-18T03:33:14.3355375Z   test_uneven_shards (__main__.TestShardedTensorEnumerable) ... skip: c10d was not compiled with the NCCL backend (0.002s)
2022-05-18T03:33:14.3374685Z   test_with_rpc_names (__main__.TestShardedTensorEnumerable) ... skip: c10d was not compiled with the NCCL backend (0.002s)
2022-05-18T03:33:14.3392643Z   test_init_from_local_shards (__main__.TestShardedTensorFromLocalShards) ... skip: c10d was not compiled with the NCCL backend (0.002s)
2022-05-18T03:33:14.3415622Z   test_init_from_local_shards_and_global_metadata (__main__.TestShardedTensorFromLocalShards) ... skip: c10d was not compiled with the NCCL backend (0.002s)
2022-05-18T03:33:14.3442486Z   test_init_from_local_shards_and_global_metadata_invalid_shards (__main__.TestShardedTensorFromLocalShards) ... skip: c10d was not compiled with the NCCL backend (0.003s)
2022-05-18T03:33:14.3455664Z   test_init_from_local_shards_invalid_local_shards (__main__.TestShardedTensorFromLocalShards) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-05-18T03:33:14.6239915Z   test_init_from_local_shards_invalid_pin_memory (__main__.TestShardedTensorFromLocalShards) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 285
2022-05-18T03:33:14.6261140Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 286
2022-05-18T03:33:14.6283830Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 287
2022-05-18T03:33:14.6307275Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 288
2022-05-18T03:33:15.3226834Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:33:15.3296207Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:33:15.3311637Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:33:15.3642076Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:33:15.3837157Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:33:15.3908384Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:33:15.3908899Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:33:15.3909657Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:33:15.3910181Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:33:15.3910750Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:33:15.3911265Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:33:15.3939500Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:33:15.6339113Z skip: Need at least 4 CUDA devices (1.288s)
2022-05-18T03:33:15.6355294Z   test_init_from_local_shards_invalid_property_cross_ranks (__main__.TestShardedTensorFromLocalShards) ... skip: c10d was not compiled with the NCCL backend (0.002s)
2022-05-18T03:33:15.6362504Z   test_init_from_local_shards_invalid_shards_gaps (__main__.TestShardedTensorFromLocalShards) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-05-18T03:33:15.6368397Z   test_init_from_local_shards_invalid_shards_overlap (__main__.TestShardedTensorFromLocalShards) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-05-18T03:33:15.6382139Z   test_init_from_local_shards_new_group (__main__.TestShardedTensorFromLocalShards) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-05-18T03:33:15.6390931Z   test_local_shards (__main__.TestShardedTensorFromLocalShards) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-05-18T03:33:15.6397193Z   test_init_from_local_tensor (__main__.TestShardedTensorFromLocalTensor) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-05-18T03:33:15.6405471Z   test_init_from_local_tensor_errors (__main__.TestShardedTensorFromLocalTensor) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-05-18T03:33:15.6781549Z   test_serialize_and_deserialize (__main__.TestShardedTensorMetadata) ... ok (0.037s)
2022-05-18T03:33:15.6781893Z 
2022-05-18T03:33:15.6782399Z ----------------------------------------------------------------------
2022-05-18T03:33:15.6782754Z Ran 58 tests in 1.395s
2022-05-18T03:33:15.6782869Z 
2022-05-18T03:33:15.6783077Z OK (skipped=57)
2022-05-18T03:33:15.6783187Z 
2022-05-18T03:33:15.6783272Z Generating XML reports...
2022-05-18T03:33:15.6815277Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardedTensorMetadata-20220518033314.xml
2022-05-18T03:33:15.6818766Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestCreateTensorFromParams-20220518033314.xml
2022-05-18T03:33:15.6822153Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestLocalTensor-20220518033314.xml
2022-05-18T03:33:15.6825820Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestModuleHookApi-20220518033314.xml
2022-05-18T03:33:15.6830255Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardParameter-20220518033314.xml
2022-05-18T03:33:15.6833971Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardTensor-20220518033314.xml
2022-05-18T03:33:15.6857575Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardedTensorChunked-20220518033314.xml
2022-05-18T03:33:15.6862310Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardedTensorCustomOps-20220518033314.xml
2022-05-18T03:33:15.6875393Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardedTensorEnumerable-20220518033314.xml
2022-05-18T03:33:15.6886717Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardedTensorFromLocalShards-20220518033314.xml
2022-05-18T03:33:15.6890506Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardedTensorFromLocalTensor-20220518033314.xml
2022-05-18T03:33:15.8763137Z Running distributed/_shard/sharded_tensor/test_sharded_tensor_reshard ... [2022-05-18 03:33:15.875920]
2022-05-18T03:33:15.8764074Z Executing ['/opt/conda/bin/python', 'distributed/_shard/sharded_tensor/test_sharded_tensor_reshard.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-05-18 03:33:15.876020]
2022-05-18T03:33:16.4453740Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor_reshard
2022-05-18T03:33:16.4463950Z 
2022-05-18T03:33:16.4464047Z Running tests...
2022-05-18T03:33:16.4464399Z ----------------------------------------------------------------------
2022-05-18T03:33:16.4472453Z   test_sharded_tensor_reshard (__main__.TestReshard) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-05-18T03:33:16.4481523Z   test_sharded_tensor_reshard_errors (__main__.TestReshard) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-05-18T03:33:16.4481944Z 
2022-05-18T03:33:16.4482806Z ----------------------------------------------------------------------
2022-05-18T03:33:16.4483293Z Ran 2 tests in 0.002s
2022-05-18T03:33:16.4483525Z 
2022-05-18T03:33:16.4483645Z OK (skipped=2)
2022-05-18T03:33:16.4483841Z 
2022-05-18T03:33:16.4483999Z Generating XML reports...
2022-05-18T03:33:16.4509461Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor_reshard/TEST-TestReshard-20220518033316.xml
2022-05-18T03:33:16.5476133Z Running distributed/_shard/sharding_plan/test_sharding_plan ... [2022-05-18 03:33:16.547166]
2022-05-18T03:33:16.5476712Z Executing ['/opt/conda/bin/python', 'distributed/_shard/sharding_plan/test_sharding_plan.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-05-18 03:33:16.547247]
2022-05-18T03:33:17.2047771Z Running distributed/_shard/sharding_spec/test_sharding_spec ... [2022-05-18 03:33:17.204333]
2022-05-18T03:33:17.2048367Z Executing ['/opt/conda/bin/python', 'distributed/_shard/sharding_spec/test_sharding_spec.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-05-18 03:33:17.204413]
2022-05-18T03:33:17.7625139Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7h3njurh
2022-05-18T03:33:17.7625893Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7h3njurh/_remote_module_non_scriptable.py
2022-05-18T03:33:17.7737846Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharding_spec.test_sharding_spec
2022-05-18T03:33:17.7751933Z 
2022-05-18T03:33:17.7752226Z Running tests...
2022-05-18T03:33:17.7752627Z ----------------------------------------------------------------------
2022-05-18T03:33:18.0532789Z   test_custom_sharding_spec (__main__.TestCustomShardingSpec) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 372
2022-05-18T03:33:18.0554101Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 373
2022-05-18T03:33:18.0576211Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 374
2022-05-18T03:33:18.0599275Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 375
2022-05-18T03:33:18.7090726Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph4d56vg6
2022-05-18T03:33:18.7091950Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph4d56vg6/_remote_module_non_scriptable.py
2022-05-18T03:33:18.7184445Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:33:18.7209811Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn6btrphp
2022-05-18T03:33:18.7210588Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9avfmd38
2022-05-18T03:33:18.7211462Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn6btrphp/_remote_module_non_scriptable.py
2022-05-18T03:33:18.7212572Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9avfmd38/_remote_module_non_scriptable.py
2022-05-18T03:33:18.7263667Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps4ywe2vy
2022-05-18T03:33:18.7265246Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps4ywe2vy/_remote_module_non_scriptable.py
2022-05-18T03:33:18.7307947Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:33:18.7308523Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:33:18.7361001Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:33:18.8627874Z ok (1.087s)
2022-05-18T03:33:18.8637426Z   test_custom_sharding_spec_shard_tensor (__main__.TestCustomShardingSpec)
2022-05-18T03:33:18.8638205Z Test custom spec can be invoked from the ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-05-18T03:33:18.8645229Z   test_custom_sharding_spec_tensor_ctor (__main__.TestCustomShardingSpec)
2022-05-18T03:33:18.8645612Z Test sharded_tensor.ones(...) with the custom ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-05-18T03:33:18.8660208Z   test_chunked_sharding_spec (__main__.TestShardingSpec) ... skip: 2 CUDA GPUs are needed (0.001s)
2022-05-18T03:33:18.8667114Z   test_device_placement (__main__.TestShardingSpec) ... skip: 2 CUDA GPUs are needed (0.001s)
2022-05-18T03:33:18.8705413Z   test_enumerable_sharding_spec (__main__.TestShardingSpec) ... skip: 2 CUDA GPUs are needed (0.004s)
2022-05-18T03:33:18.8725058Z   test_get_chunk_sharding_params (__main__.TestShardingSpec) ... ok (0.002s)
2022-05-18T03:33:18.8734957Z   test_get_chunked_dim_size (__main__.TestShardingSpec) ... ok (0.001s)
2022-05-18T03:33:18.8745062Z   test_get_split_size (__main__.TestShardingSpec) ... ok (0.001s)
2022-05-18T03:33:18.8822234Z   test_infer_sharding_spec_from_shards_metadata (__main__.TestShardingSpec) ... ok (0.008s)
2022-05-18T03:33:18.8823049Z 
2022-05-18T03:33:18.8823576Z ----------------------------------------------------------------------
2022-05-18T03:33:18.8823933Z Ran 10 tests in 1.107s
2022-05-18T03:33:18.8824089Z 
2022-05-18T03:33:18.8824184Z OK (skipped=5)
2022-05-18T03:33:18.8824293Z 
2022-05-18T03:33:18.8824382Z Generating XML reports...
2022-05-18T03:33:18.8867343Z Generated XML report: test-reports/python-unittest/distributed._shard.sharding_spec.test_sharding_spec/TEST-TestCustomShardingSpec-20220518033317.xml
2022-05-18T03:33:18.8875600Z Generated XML report: test-reports/python-unittest/distributed._shard.sharding_spec.test_sharding_spec/TEST-TestShardingSpec-20220518033317.xml
2022-05-18T03:33:19.0746038Z Running distributed/_shard/test_partial_tensor ... [2022-05-18 03:33:19.074223]
2022-05-18T03:33:19.0746887Z Executing ['/opt/conda/bin/python', 'distributed/_shard/test_partial_tensor.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-05-18 03:33:19.074323]
2022-05-18T03:33:19.6444792Z Test results will be stored in test-reports/python-unittest/distributed._shard.test_partial_tensor
2022-05-18T03:33:19.6456126Z 
2022-05-18T03:33:19.6456224Z Running tests...
2022-05-18T03:33:19.6456896Z ----------------------------------------------------------------------
2022-05-18T03:33:19.6468207Z   test_cat (__main__.TestPartialTensorOps) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-05-18T03:33:19.6475937Z   test_cat_errors (__main__.TestPartialTensorOps) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-05-18T03:33:19.6480235Z   test_transpose (__main__.TestPartialTensorOps) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-05-18T03:33:19.6486858Z   test_partial_tensor_reshard (__main__.TestPartialTensorReshard) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-05-18T03:33:19.6497704Z   test_partial_tensor_reshard_errors (__main__.TestPartialTensorReshard) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-05-18T03:33:19.6498168Z 
2022-05-18T03:33:19.6498622Z ----------------------------------------------------------------------
2022-05-18T03:33:19.6499052Z Ran 5 tests in 0.004s
2022-05-18T03:33:19.6499261Z 
2022-05-18T03:33:19.6499561Z OK (skipped=5)
2022-05-18T03:33:19.6499669Z 
2022-05-18T03:33:19.6499755Z Generating XML reports...
2022-05-18T03:33:19.6524750Z Generated XML report: test-reports/python-unittest/distributed._shard.test_partial_tensor/TEST-TestPartialTensorOps-20220518033319.xml
2022-05-18T03:33:19.6528237Z Generated XML report: test-reports/python-unittest/distributed._shard.test_partial_tensor/TEST-TestPartialTensorReshard-20220518033319.xml
2022-05-18T03:33:19.7486116Z Running distributed/_shard/test_replicated_tensor ... [2022-05-18 03:33:19.748185]
2022-05-18T03:33:19.7486698Z Executing ['/opt/conda/bin/python', 'distributed/_shard/test_replicated_tensor.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-05-18 03:33:19.748259]
2022-05-18T03:33:20.4087333Z Running distributed/_shard/test_sharder ... [2022-05-18 03:33:20.408354]
2022-05-18T03:33:20.4087929Z Executing ['/opt/conda/bin/python', 'distributed/_shard/test_sharder.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-05-18 03:33:20.408429]
2022-05-18T03:33:20.9625752Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2u93o8i_
2022-05-18T03:33:20.9627054Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2u93o8i_/_remote_module_non_scriptable.py
2022-05-18T03:33:21.0656793Z Running distributed/algorithms/test_join ... [2022-05-18 03:33:21.065340]
2022-05-18T03:33:21.0657357Z Executing ['/opt/conda/bin/python', 'distributed/algorithms/test_join.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-05-18 03:33:21.065414]
2022-05-18T03:33:21.6167347Z Test results will be stored in test-reports/python-unittest/distributed.algorithms.test_join
2022-05-18T03:33:21.6179094Z 
2022-05-18T03:33:21.6179353Z Running tests...
2022-05-18T03:33:21.6179969Z ----------------------------------------------------------------------
2022-05-18T03:33:21.6188147Z   test_join_kwargs (__main__.TestJoin)
2022-05-18T03:33:21.8969227Z Tests passing keyword arguments to the context manager. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 457
2022-05-18T03:33:21.8990729Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 458
2022-05-18T03:33:22.4547403Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:33:22.4552077Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:33:22.4658911Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:33:22.4659607Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:33:22.4660349Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:33:22.4660876Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:33:22.6013227Z ok (0.983s)
2022-05-18T03:33:22.6020104Z   test_multiple_joinable_disable (__main__.TestJoin)
2022-05-18T03:33:22.6054528Z Tests ``enable=False`` for multiple :class:`Joinable` s. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 485
2022-05-18T03:33:22.6079536Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 486
2022-05-18T03:33:23.1571025Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:33:23.1586094Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:33:23.1692843Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:33:23.1693393Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:33:23.1694013Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:33:23.1694699Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:33:23.3098564Z ok (0.708s)
2022-05-18T03:33:23.3107055Z   test_multiple_joinables (__main__.TestJoin)
2022-05-18T03:33:23.3142384Z Tests the main hooks and post-hooks of multiple :class:`Joinable` s ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 513
2022-05-18T03:33:23.3167438Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 514
2022-05-18T03:33:23.8693251Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:33:23.9051296Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:33:23.9202576Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:33:23.9202984Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:33:23.9203634Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:33:23.9204156Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:33:24.1188644Z ok (0.809s)
2022-05-18T03:33:24.1194367Z   test_multiple_joinables_throw (__main__.TestJoin)
2022-05-18T03:33:24.1229650Z Tests ``throw_on_early_termination=True`` for multiple ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 541
2022-05-18T03:33:24.1254879Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 542
2022-05-18T03:33:24.6807286Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:33:24.6807895Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:33:24.7015727Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:33:24.7016334Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:33:24.7016968Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:33:24.7017498Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:33:24.8275137Z ok (0.709s)
2022-05-18T03:33:24.8282582Z   test_single_joinable (__main__.TestJoin)
2022-05-18T03:33:24.8317251Z Tests the main hooks and post-hooks of a single :class:`Joinable` ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 569
2022-05-18T03:33:24.8342113Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 570
2022-05-18T03:33:25.3896159Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:33:25.4160307Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:33:25.4306185Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:33:25.4306570Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:33:25.4307265Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:33:25.4307863Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:33:25.6363096Z ok (0.809s)
2022-05-18T03:33:25.6369662Z   test_single_joinable_disable (__main__.TestJoin)
2022-05-18T03:33:25.6404760Z Tests ``enable=False`` for a single :class:`Joinable`. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 597
2022-05-18T03:33:25.6429486Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 598
2022-05-18T03:33:26.1992441Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:33:26.2136580Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:33:26.2300112Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:33:26.2300737Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:33:26.2301653Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:33:26.2302467Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:33:26.4450563Z ok (0.809s)
2022-05-18T03:33:26.4458002Z   test_single_joinable_main_hooks (__main__.TestJoin)
2022-05-18T03:33:26.4493770Z Tests the main hooks of a single :class:`Joinable`. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 625
2022-05-18T03:33:26.4518389Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 626
2022-05-18T03:33:27.0065835Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:33:27.0273239Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:33:27.0481681Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:33:27.0482081Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:33:27.0482698Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:33:27.0483229Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:33:27.2539854Z ok (0.809s)
2022-05-18T03:33:27.2545753Z   test_single_joinable_post_hooks (__main__.TestJoin)
2022-05-18T03:33:27.2581396Z Tests the post-hooks of a single :class:`Joinable`. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 653
2022-05-18T03:33:27.2606878Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 654
2022-05-18T03:33:27.8216546Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:33:27.8219094Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:33:27.8423699Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:33:27.8424335Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:33:27.8425257Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:33:27.8426111Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:33:28.0632680Z ok (0.809s)
2022-05-18T03:33:28.0638465Z   test_single_joinable_throw (__main__.TestJoin)
2022-05-18T03:33:28.0673745Z Tests ``throw_on_early_termination=True`` for a single ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 681
2022-05-18T03:33:28.0699712Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 682
2022-05-18T03:33:28.6270019Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:33:28.6493032Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:33:28.6678161Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:33:28.6678864Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:33:28.6679644Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:33:28.6680239Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:33:28.8722600Z ok (0.809s)
2022-05-18T03:33:28.8722747Z 
2022-05-18T03:33:28.8723184Z ----------------------------------------------------------------------
2022-05-18T03:33:28.8723445Z Ran 9 tests in 7.254s
2022-05-18T03:33:28.8723607Z 
2022-05-18T03:33:28.8723705Z OK
2022-05-18T03:33:28.8723811Z 
2022-05-18T03:33:28.8723908Z Generating XML reports...
2022-05-18T03:33:28.8762748Z Generated XML report: test-reports/python-unittest/distributed.algorithms.test_join/TEST-TestJoin-20220518033321.xml
2022-05-18T03:33:29.0618345Z Running distributed/elastic/events/lib_test ... [2022-05-18 03:33:29.061408]
2022-05-18T03:33:29.0618832Z Executing ['/opt/conda/bin/python', '-m', 'pytest', 'distributed/elastic/events/lib_test.py', '-v'] ... [2022-05-18 03:33:29.061507]
2022-05-18T03:33:29.6526137Z ============================= test session starts ==============================
2022-05-18T03:33:29.6526607Z platform linux -- Python 3.7.13, pytest-7.1.2, pluggy-1.0.0 -- /opt/conda/bin/python
2022-05-18T03:33:29.6564301Z cachedir: .pytest_cache
2022-05-18T03:33:29.6565015Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2022-05-18T03:33:29.6565509Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2022-05-18T03:33:29.6565799Z plugins: hypothesis-4.53.2
2022-05-18T03:33:30.1751384Z collecting ... 
2022-05-18T03:33:30.1759975Z collecting 3 items                                                             
2022-05-18T03:33:30.1760525Z collected 8 items                                                              
2022-05-18T03:33:30.1764496Z 
2022-05-18T03:33:30.1777430Z distributed/elastic/events/lib_test.py::EventLibTest::test_event_created PASSED [ 12%]
2022-05-18T03:33:30.1787845Z distributed/elastic/events/lib_test.py::EventLibTest::test_event_deser PASSED [ 25%]
2022-05-18T03:33:30.1800408Z distributed/elastic/events/lib_test.py::EventLibTest::test_get_or_create_logger PASSED [ 37%]
2022-05-18T03:33:30.2286407Z distributed/elastic/events/lib_test.py::RdzvEventLibTest::test_construct_and_record_rdzv_event PASSED [ 50%]
2022-05-18T03:33:30.2299902Z distributed/elastic/events/lib_test.py::RdzvEventLibTest::test_construct_and_record_rdzv_event_does_not_run_if_invalid_dest PASSED [ 62%]
2022-05-18T03:33:30.2309211Z distributed/elastic/events/lib_test.py::RdzvEventLibTest::test_rdzv_event_created PASSED [ 75%]
2022-05-18T03:33:30.2319653Z distributed/elastic/events/lib_test.py::RdzvEventLibTest::test_rdzv_event_deserialize PASSED [ 87%]
2022-05-18T03:33:30.2334083Z distributed/elastic/events/lib_test.py::RdzvEventLibTest::test_rdzv_event_str PASSED [100%]
2022-05-18T03:33:30.2335245Z 
2022-05-18T03:33:30.2335640Z ============================== 8 passed in 0.58s ===============================
2022-05-18T03:33:30.3604067Z Running distributed/elastic/metrics/api_test ... [2022-05-18 03:33:30.360042]
2022-05-18T03:33:30.3604722Z Executing ['/opt/conda/bin/python', 'distributed/elastic/metrics/api_test.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-05-18 03:33:30.360121]
2022-05-18T03:33:30.9098176Z Test results will be stored in test-reports/python-unittest/distributed.elastic.metrics.api_test
2022-05-18T03:33:30.9109007Z 
2022-05-18T03:33:30.9109108Z Running tests...
2022-05-18T03:33:30.9109841Z ----------------------------------------------------------------------
2022-05-18T03:33:31.1849594Z   test_get_metric_name (__main__.MetricsApiTest) ... ok (0.274s)
2022-05-18T03:33:31.1860838Z   test_inheritance (__main__.MetricsApiTest) ... ok (0.001s)
2022-05-18T03:33:31.1876597Z   test_profile (__main__.MetricsApiTest) ... ok (0.001s)
2022-05-18T03:33:31.1876888Z 
2022-05-18T03:33:31.1877652Z ----------------------------------------------------------------------
2022-05-18T03:33:31.1877950Z Ran 3 tests in 0.277s
2022-05-18T03:33:31.1878068Z 
2022-05-18T03:33:31.1878117Z OK
2022-05-18T03:33:31.1879029Z 
2022-05-18T03:33:31.1879263Z Generating XML reports...
2022-05-18T03:33:31.1904277Z Generated XML report: test-reports/python-unittest/distributed.elastic.metrics.api_test/TEST-MetricsApiTest-20220518033330.xml
2022-05-18T03:33:31.3586806Z Running distributed/elastic/multiprocessing/api_test ... [2022-05-18 03:33:31.358227]
2022-05-18T03:33:31.3587441Z Executing ['/opt/conda/bin/python', 'distributed/elastic/multiprocessing/api_test.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-05-18 03:33:31.358320]
2022-05-18T03:33:31.9642573Z Test results will be stored in test-reports/python-unittest/distributed.elastic.multiprocessing.api_test
2022-05-18T03:33:31.9657543Z 
2022-05-18T03:33:31.9657633Z Running tests...
2022-05-18T03:33:31.9658206Z ----------------------------------------------------------------------
2022-05-18T03:33:32.2408967Z   test_get_failures (__main__.RunProcResultsTest) ... ok (0.275s)
2022-05-18T03:33:32.2418348Z   test_is_failed (__main__.RunProcResultsTest) ... ok (0.001s)
2022-05-18T03:33:32.2433363Z   test_args_env_len_mismatch (__main__.StartProcessesListTest) ... ok (0.001s)
2022-05-18T03:33:32.2684057Z   test_binary (__main__.StartProcessesListTest) ... hello stdout from 0
2022-05-18T03:33:32.2684349Z hello stderr from 0
2022-05-18T03:33:32.2709908Z hello stdout from 1
2022-05-18T03:33:32.2710168Z hello stderr from 1
2022-05-18T03:33:32.3533907Z ok (0.110s)
2022-05-18T03:33:32.3814362Z   test_binary_exit (__main__.StartProcessesListTest) ... bar stdout from 1
2022-05-18T03:33:32.3814661Z bar stderr from 1
2022-05-18T03:33:32.4631274Z failed (exitcode: 138) local_rank: 0 (pid: 741) of binary: distributed/elastic/multiprocessing/bin/echo1.py
2022-05-18T03:33:32.4641586Z ok (0.111s)
2022-05-18T03:33:32.4700419Z   test_binary_incorrect_entrypoint (__main__.StartProcessesListTest) ... ok (0.006s)
2022-05-18T03:33:32.4941003Z   test_binary_raises (__main__.StartProcessesListTest) ... Traceback (most recent call last):
2022-05-18T03:33:32.4941427Z   File "distributed/elastic/multiprocessing/bin/echo2.py", line 22, in <module>
2022-05-18T03:33:32.4941724Z     raise RuntimeError(f"raised from {rank}")
2022-05-18T03:33:32.4941932Z RuntimeError: raised from 0
2022-05-18T03:33:32.4963921Z bar from 1
2022-05-18T03:33:32.5781418Z failed (exitcode: 1) local_rank: 0 (pid: 744) of binary: distributed/elastic/multiprocessing/bin/echo2.py
2022-05-18T03:33:32.5794627Z ok (0.109s)
2022-05-18T03:33:32.6071983Z   test_binary_redirect_and_tee (__main__.StartProcessesListTest) ... world stdout from 1
2022-05-18T03:33:32.6891538Z [trainer0]:hello stdout from 0
2022-05-18T03:33:32.6891795Z [trainer1]:world stderr from 1
2022-05-18T03:33:33.6915470Z ok (1.112s)
2022-05-18T03:33:34.3288164Z   test_function (__main__.StartProcessesListTest) ... hello stdout from 0
2022-05-18T03:33:34.3288535Z hello stderr from 0
2022-05-18T03:33:34.3686932Z hello stdout from 1
2022-05-18T03:33:34.3687134Z hello stderr from 1
2022-05-18T03:33:34.5148033Z Closing process 752 via signal SIGTERM
2022-05-18T03:33:34.5174273Z ok (0.826s)
2022-05-18T03:33:35.6315840Z   test_function_large_ret_val (__main__.StartProcessesListTest) ... Closing process 771 via signal SIGTERM
2022-05-18T03:33:35.6316336Z Closing process 772 via signal SIGTERM
2022-05-18T03:33:35.6316561Z Closing process 774 via signal SIGTERM
2022-05-18T03:33:35.6347680Z ok (1.117s)
2022-05-18T03:33:35.6365459Z   test_function_raise (__main__.StartProcessesListTest)
2022-05-18T03:33:36.4841385Z run 2x copies of echo2, raise an exception on the first ... failed (exitcode: 1) local_rank: 0 (pid: 811) of fn: echo2 (start_method: spawn)
2022-05-18T03:33:36.4842307Z Traceback (most recent call last):
2022-05-18T03:33:36.4843229Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/elastic/multiprocessing/api.py", line 453, in _poll
2022-05-18T03:33:36.4843714Z     self._pc.join(-1)
2022-05-18T03:33:36.4844061Z   File "/opt/conda/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 160, in join
2022-05-18T03:33:36.4844407Z     raise ProcessRaisedException(msg, error_index, failed_process.pid)
2022-05-18T03:33:36.4844746Z torch.multiprocessing.spawn.ProcessRaisedException: 
2022-05-18T03:33:36.4844942Z 
2022-05-18T03:33:36.4845105Z -- Process 0 terminated with the following error:
2022-05-18T03:33:36.4845328Z Traceback (most recent call last):
2022-05-18T03:33:36.4845693Z   File "/opt/conda/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 69, in _wrap
2022-05-18T03:33:36.4845952Z     fn(i, *args)
2022-05-18T03:33:36.4846315Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/elastic/multiprocessing/api.py", line 369, in _wrap
2022-05-18T03:33:36.4846603Z     ret = record(fn)(*args_)
2022-05-18T03:33:36.4847006Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 345, in wrapper
2022-05-18T03:33:36.4847291Z     return f(*args, **kwargs)
2022-05-18T03:33:36.4847579Z   File "/var/lib/jenkins/workspace/test/distributed/elastic/multiprocessing/api_test.py", line 138, in echo2
2022-05-18T03:33:36.4847857Z     raise RuntimeError(msg)
2022-05-18T03:33:36.4848048Z RuntimeError: hello
2022-05-18T03:33:36.4848148Z 
2022-05-18T03:33:36.4856048Z ok (0.851s)
2022-05-18T03:33:36.4876481Z   test_function_with_tensor (__main__.StartProcessesListTest) ... ok (0.002s)
2022-05-18T03:33:36.4889668Z   test_invalid_log_dir (__main__.StartProcessesListTest) ... ok (0.001s)
2022-05-18T03:33:36.4932353Z   test_multiprocess_context_close (__main__.StartProcessesListTest) ... Closing process 831 via signal SIGTERM
2022-05-18T03:33:36.4945693Z ok (0.006s)
2022-05-18T03:33:36.4978704Z   test_multiprocessing_context_poll_raises_exception (__main__.StartProcessesListTest) ... failed (exitcode: -1) local_rank: 0 (pid: 123) of fn: echo0 (start_method: spawn)
2022-05-18T03:33:36.4979378Z Traceback (most recent call last):
2022-05-18T03:33:36.4979819Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/elastic/multiprocessing/api.py", line 453, in _poll
2022-05-18T03:33:36.4980142Z     self._pc.join(-1)
2022-05-18T03:33:36.4980384Z   File "/opt/conda/lib/python3.7/unittest/mock.py", line 1016, in __call__
2022-05-18T03:33:36.4980647Z     return _mock_self._mock_call(*args, **kwargs)
2022-05-18T03:33:36.4980900Z   File "/opt/conda/lib/python3.7/unittest/mock.py", line 1076, in _mock_call
2022-05-18T03:33:36.4981129Z     raise effect
2022-05-18T03:33:36.4981401Z torch.multiprocessing.spawn.ProcessRaisedException: test msg
2022-05-18T03:33:36.4989087Z ok (0.004s)
2022-05-18T03:33:38.3023114Z   test_pcontext_wait (__main__.StartProcessesListTest) ... ok (1.803s)
2022-05-18T03:33:38.3078014Z   test_subprocess_context_close (__main__.StartProcessesListTest) ... Sending process 842 closing signal SIGTERM
2022-05-18T03:33:38.3092876Z ok (0.007s)
2022-05-18T03:33:38.3113195Z   test_to_map (__main__.StartProcessesListTest) ... ok (0.002s)
2022-05-18T03:33:38.3120562Z   test_validate_full_rank (__main__.StartProcessesListTest) ... ok (0.001s)
2022-05-18T03:33:38.9615648Z   test_void_function (__main__.StartProcessesListTest) ... world
2022-05-18T03:33:38.9880549Z hello
2022-05-18T03:33:39.1266677Z Closing process 844 via signal SIGTERM
2022-05-18T03:33:39.1291148Z ok (0.817s)
2022-05-18T03:33:39.1311992Z   test_args_env_len_mismatch (__main__.StartProcessesTest) ... ok (0.002s)
2022-05-18T03:33:39.1596159Z   test_binary_exit (__main__.StartProcessesTest) ... bar stdout from 1
2022-05-18T03:33:39.1596604Z bar stderr from 1
2022-05-18T03:33:39.2411224Z failed (exitcode: 138) local_rank: 0 (pid: 863) of binary: distributed/elastic/multiprocessing/bin/echo1.py
2022-05-18T03:33:39.2421721Z ok (0.111s)
2022-05-18T03:33:39.2480867Z   test_binary_incorrect_entrypoint (__main__.StartProcessesTest) ... ok (0.006s)
2022-05-18T03:33:39.2721199Z   test_binary_raises (__main__.StartProcessesTest) ... Traceback (most recent call last):
2022-05-18T03:33:39.2721824Z   File "distributed/elastic/multiprocessing/bin/echo2.py", line 22, in <module>
2022-05-18T03:33:39.2722148Z     raise RuntimeError(f"raised from {rank}")
2022-05-18T03:33:39.2722369Z RuntimeError: raised from 0
2022-05-18T03:33:39.2746976Z bar from 1
2022-05-18T03:33:39.3562756Z failed (exitcode: 1) local_rank: 0 (pid: 866) of binary: distributed/elastic/multiprocessing/bin/echo2.py
2022-05-18T03:33:39.3570977Z ok (0.109s)
2022-05-18T03:33:40.4505106Z   test_function_large_ret_val (__main__.StartProcessesTest) ... Closing process 868 via signal SIGTERM
2022-05-18T03:33:40.4505656Z Closing process 869 via signal SIGTERM
2022-05-18T03:33:40.4506040Z Closing process 870 via signal SIGTERM
2022-05-18T03:33:40.4523617Z ok (1.095s)
2022-05-18T03:33:40.4538238Z   test_function_raise (__main__.StartProcessesTest)
2022-05-18T03:33:41.3016412Z run 2x copies of echo2, raise an exception on the first ... failed (exitcode: 1) local_rank: 0 (pid: 908) of fn: echo2 (start_method: spawn)
2022-05-18T03:33:41.3016983Z Traceback (most recent call last):
2022-05-18T03:33:41.3017901Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/elastic/multiprocessing/api.py", line 453, in _poll
2022-05-18T03:33:41.3018397Z     self._pc.join(-1)
2022-05-18T03:33:41.3018847Z   File "/opt/conda/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 160, in join
2022-05-18T03:33:41.3019310Z     raise ProcessRaisedException(msg, error_index, failed_process.pid)
2022-05-18T03:33:41.3019872Z torch.multiprocessing.spawn.ProcessRaisedException: 
2022-05-18T03:33:41.3020246Z 
2022-05-18T03:33:41.3020478Z -- Process 0 terminated with the following error:
2022-05-18T03:33:41.3020703Z Traceback (most recent call last):
2022-05-18T03:33:41.3021088Z   File "/opt/conda/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 69, in _wrap
2022-05-18T03:33:41.3021353Z     fn(i, *args)
2022-05-18T03:33:41.3021734Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/elastic/multiprocessing/api.py", line 369, in _wrap
2022-05-18T03:33:41.3022010Z     ret = record(fn)(*args_)
2022-05-18T03:33:41.3022415Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 345, in wrapper
2022-05-18T03:33:41.3022717Z     return f(*args, **kwargs)
2022-05-18T03:33:41.3023174Z   File "/var/lib/jenkins/workspace/test/distributed/elastic/multiprocessing/api_test.py", line 138, in echo2
2022-05-18T03:33:41.3023458Z     raise RuntimeError(msg)
2022-05-18T03:33:41.3023655Z RuntimeError: hello
2022-05-18T03:33:41.3023776Z 
2022-05-18T03:33:41.3032069Z ok (0.851s)
2022-05-18T03:33:41.3052479Z   test_function_with_tensor (__main__.StartProcessesTest) ... ok (0.002s)
2022-05-18T03:33:41.3066113Z   test_invalid_log_dir (__main__.StartProcessesTest) ... ok (0.001s)
2022-05-18T03:33:41.3109689Z   test_multiprocess_context_close (__main__.StartProcessesTest) ... Closing process 928 via signal SIGTERM
2022-05-18T03:33:41.3123221Z ok (0.006s)
2022-05-18T03:33:41.3151367Z   test_multiprocessing_context_poll_raises_exception (__main__.StartProcessesTest) ... failed (exitcode: -1) local_rank: 0 (pid: 123) of fn: echo0 (start_method: spawn)
2022-05-18T03:33:41.3151762Z Traceback (most recent call last):
2022-05-18T03:33:41.3152188Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/elastic/multiprocessing/api.py", line 453, in _poll
2022-05-18T03:33:41.3152495Z     self._pc.join(-1)
2022-05-18T03:33:41.3152737Z   File "/opt/conda/lib/python3.7/unittest/mock.py", line 1016, in __call__
2022-05-18T03:33:41.3153001Z     return _mock_self._mock_call(*args, **kwargs)
2022-05-18T03:33:41.3153255Z   File "/opt/conda/lib/python3.7/unittest/mock.py", line 1076, in _mock_call
2022-05-18T03:33:41.3153487Z     raise effect
2022-05-18T03:33:41.3153924Z torch.multiprocessing.spawn.ProcessRaisedException: test msg
2022-05-18T03:33:41.3161223Z ok (0.004s)
2022-05-18T03:33:43.1181859Z   test_pcontext_wait (__main__.StartProcessesTest) ... ok (1.802s)
2022-05-18T03:33:43.1238129Z   test_subprocess_context_close (__main__.StartProcessesTest) ... Sending process 939 closing signal SIGTERM
2022-05-18T03:33:43.1252926Z ok (0.007s)
2022-05-18T03:33:43.1271795Z   test_to_map (__main__.StartProcessesTest) ... ok (0.002s)
2022-05-18T03:33:43.1278756Z   test_validate_full_rank (__main__.StartProcessesTest) ... ok (0.001s)
2022-05-18T03:33:43.7574308Z   test_void_function (__main__.StartProcessesTest) ... hello
2022-05-18T03:33:43.8016243Z world
2022-05-18T03:33:43.9419553Z Closing process 941 via signal SIGTERM
2022-05-18T03:33:43.9429930Z ok (0.815s)
2022-05-18T03:33:43.9446445Z   test_from_str_bad_input (__main__.StdTest) ... ok (0.001s)
2022-05-18T03:33:43.9456765Z   test_from_value (__main__.StdTest) ... ok (0.001s)
2022-05-18T03:33:43.9465738Z   test_from_value_map (__main__.StdTest) ... ok (0.001s)
2022-05-18T03:33:43.9465994Z 
2022-05-18T03:33:43.9466502Z ----------------------------------------------------------------------
2022-05-18T03:33:43.9466903Z Ran 38 tests in 11.981s
2022-05-18T03:33:43.9467019Z 
2022-05-18T03:33:43.9467080Z OK
2022-05-18T03:33:43.9467158Z 
2022-05-18T03:33:43.9467249Z Generating XML reports...
2022-05-18T03:33:43.9515905Z Generated XML report: test-reports/python-unittest/distributed.elastic.multiprocessing.api_test/TEST-RunProcResultsTest-20220518033331.xml
2022-05-18T03:33:43.9534194Z Generated XML report: test-reports/python-unittest/distributed.elastic.multiprocessing.api_test/TEST-StartProcessesListTest-20220518033331.xml
2022-05-18T03:33:43.9549579Z Generated XML report: test-reports/python-unittest/distributed.elastic.multiprocessing.api_test/TEST-StartProcessesTest-20220518033331.xml
2022-05-18T03:33:43.9554392Z Generated XML report: test-reports/python-unittest/distributed.elastic.multiprocessing.api_test/TEST-StdTest-20220518033331.xml
2022-05-18T03:33:44.1382471Z Running distributed/elastic/timer/api_test ... [2022-05-18 03:33:44.137850]
2022-05-18T03:33:44.1383330Z Executing ['/opt/conda/bin/python', 'distributed/elastic/timer/api_test.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-05-18 03:33:44.137931]
2022-05-18T03:33:44.7334742Z Running distributed/elastic/timer/local_timer_example ... [2022-05-18 03:33:44.733082]
2022-05-18T03:33:44.7335327Z Executing ['/opt/conda/bin/python', 'distributed/elastic/timer/local_timer_example.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-05-18 03:33:44.733157]
2022-05-18T03:33:45.2841478Z Test results will be stored in test-reports/python-unittest/distributed.elastic.timer.local_timer_example
2022-05-18T03:33:45.2851308Z 
2022-05-18T03:33:45.2851630Z Running tests...
2022-05-18T03:33:45.2852325Z ----------------------------------------------------------------------
2022-05-18T03:33:45.5643931Z   test_example_start_method_spawn (__main__.LocalTimerExample) ... [INFO] 2022-05-18 03:33:45,563 api: Starting LocalTimerServer... max_interval=0.01, daemon=True
2022-05-18T03:33:45.5644396Z [INFO] 2022-05-18 03:33:45,564 api: Starting watchdog thread...
2022-05-18T03:33:46.5370792Z [INFO] 2022-05-18 03:33:46,536 api: Timer client configured to: LocalTimerClient
2022-05-18T03:33:46.5373432Z [INFO] 2022-05-18 03:33:46,536 api: Timer client configured to: LocalTimerClient
2022-05-18T03:33:46.5481505Z [INFO] 2022-05-18 03:33:46,547 api: Timer client configured to: LocalTimerClient
2022-05-18T03:33:46.5486196Z [INFO] 2022-05-18 03:33:46,548 api: Timer client configured to: LocalTimerClient
2022-05-18T03:33:46.5583507Z [INFO] 2022-05-18 03:33:46,557 api: Timer client configured to: LocalTimerClient
2022-05-18T03:33:46.5634654Z [INFO] 2022-05-18 03:33:46,563 api: Timer client configured to: LocalTimerClient
2022-05-18T03:33:46.5849978Z [INFO] 2022-05-18 03:33:46,584 api: Timer client configured to: LocalTimerClient
2022-05-18T03:33:46.5856526Z [INFO] 2022-05-18 03:33:46,585 api: Timer client configured to: LocalTimerClient
2022-05-18T03:33:47.6107602Z [INFO] 2022-05-18 03:33:47,610 api: Reaping worker_id=[982]. Expired timers: ['/opt/conda/lib/python3.7/contextlib.py#112']
2022-05-18T03:33:47.6108042Z [INFO] 2022-05-18 03:33:47,610 api: Successfully reaped worker=[982]
2022-05-18T03:33:47.6209912Z [INFO] 2022-05-18 03:33:47,620 api: Reaping worker_id=[984]. Expired timers: ['/opt/conda/lib/python3.7/contextlib.py#112']
2022-05-18T03:33:47.6211146Z [INFO] 2022-05-18 03:33:47,620 api: Successfully reaped worker=[984]
2022-05-18T03:33:47.6211897Z [INFO] 2022-05-18 03:33:47,620 api: Reaping worker_id=[988]. Expired timers: ['/opt/conda/lib/python3.7/contextlib.py#112']
2022-05-18T03:33:47.6212513Z [INFO] 2022-05-18 03:33:47,620 api: Successfully reaped worker=[988]
2022-05-18T03:33:47.6414524Z [INFO] 2022-05-18 03:33:47,641 api: Reaping worker_id=[986]. Expired timers: ['/opt/conda/lib/python3.7/contextlib.py#112']
2022-05-18T03:33:47.6415195Z [INFO] 2022-05-18 03:33:47,641 api: Successfully reaped worker=[986]
2022-05-18T03:33:47.6452735Z [INFO] 2022-05-18 03:33:47,645 api: Stopping LocalTimerServer
2022-05-18T03:33:47.6453256Z [INFO] 2022-05-18 03:33:47,645 api: Stopping watchdog thread...
2022-05-18T03:33:47.6520875Z ok (2.367s)
2022-05-18T03:33:47.6538929Z   test_torch_mp_example (__main__.LocalTimerExample) ... [INFO] 2022-05-18 03:33:47,653 api: Starting LocalTimerServer... max_interval=0.01, daemon=True
2022-05-18T03:33:47.6539369Z [INFO] 2022-05-18 03:33:47,653 api: Starting watchdog thread...
2022-05-18T03:33:48.5806041Z [INFO] 2022-05-18 03:33:48,579 api: Timer client configured to: LocalTimerClient
2022-05-18T03:33:48.5967246Z [INFO] 2022-05-18 03:33:48,596 api: Timer client configured to: LocalTimerClient
2022-05-18T03:33:48.6101148Z [INFO] 2022-05-18 03:33:48,609 api: Timer client configured to: LocalTimerClient
2022-05-18T03:33:48.6252719Z [INFO] 2022-05-18 03:33:48,624 api: Timer client configured to: LocalTimerClient
2022-05-18T03:33:48.6303448Z [INFO] 2022-05-18 03:33:48,629 api: Timer client configured to: LocalTimerClient
2022-05-18T03:33:48.6341739Z [INFO] 2022-05-18 03:33:48,633 api: Timer client configured to: LocalTimerClient
2022-05-18T03:33:48.6396197Z [INFO] 2022-05-18 03:33:48,639 api: Timer client configured to: LocalTimerClient
2022-05-18T03:33:48.6688440Z [INFO] 2022-05-18 03:33:48,668 api: Timer client configured to: LocalTimerClient
2022-05-18T03:33:50.3614086Z [INFO] 2022-05-18 03:33:50,360 api: Timer client configured to: LocalTimerClient
2022-05-18T03:33:50.3848899Z [INFO] 2022-05-18 03:33:50,384 api: Timer client configured to: LocalTimerClient
2022-05-18T03:33:50.3890607Z [INFO] 2022-05-18 03:33:50,388 api: Timer client configured to: LocalTimerClient
2022-05-18T03:33:50.4104925Z [INFO] 2022-05-18 03:33:50,409 api: Timer client configured to: LocalTimerClient
2022-05-18T03:33:50.4175417Z [INFO] 2022-05-18 03:33:50,417 api: Timer client configured to: LocalTimerClient
2022-05-18T03:33:50.4187230Z [INFO] 2022-05-18 03:33:50,418 api: Timer client configured to: LocalTimerClient
2022-05-18T03:33:50.4280172Z [INFO] 2022-05-18 03:33:50,427 api: Timer client configured to: LocalTimerClient
2022-05-18T03:33:50.4316295Z [INFO] 2022-05-18 03:33:50,431 api: Timer client configured to: LocalTimerClient
2022-05-18T03:33:51.4371093Z [INFO] 2022-05-18 03:33:51,436 api: Reaping worker_id=[1162]. Expired timers: ['/opt/conda/lib/python3.7/contextlib.py#112']
2022-05-18T03:33:51.4371586Z [INFO] 2022-05-18 03:33:51,436 api: Successfully reaped worker=[1162]
2022-05-18T03:33:51.4575017Z [INFO] 2022-05-18 03:33:51,457 api: Reaping worker_id=[1165]. Expired timers: ['/opt/conda/lib/python3.7/contextlib.py#112']
2022-05-18T03:33:51.4575714Z [INFO] 2022-05-18 03:33:51,457 api: Successfully reaped worker=[1165]
2022-05-18T03:33:51.4655890Z [INFO] 2022-05-18 03:33:51,465 api: Stopping LocalTimerServer
2022-05-18T03:33:51.4656250Z [INFO] 2022-05-18 03:33:51,465 api: Stopping watchdog thread...
2022-05-18T03:33:51.4676632Z [INFO] 2022-05-18 03:33:51,467 api: Reaping worker_id=[1164]. Expired timers: ['/opt/conda/lib/python3.7/contextlib.py#112']
2022-05-18T03:33:51.4677320Z [INFO] 2022-05-18 03:33:51,467 local_timer: Process with pid=1164 does not exist. Skipping
2022-05-18T03:33:51.4677735Z [INFO] 2022-05-18 03:33:51,467 api: Successfully reaped worker=[1164]
2022-05-18T03:33:51.4681036Z ok (3.816s)
2022-05-18T03:33:51.4683032Z 
2022-05-18T03:33:51.4683552Z ----------------------------------------------------------------------
2022-05-18T03:33:51.4683994Z Ran 2 tests in 6.183s
2022-05-18T03:33:51.4684125Z 
2022-05-18T03:33:51.4684191Z OK
2022-05-18T03:33:51.4684282Z 
2022-05-18T03:33:51.4684367Z Generating XML reports...
2022-05-18T03:33:51.4716062Z Generated XML report: test-reports/python-unittest/distributed.elastic.timer.local_timer_example/TEST-LocalTimerExample-20220518033345.xml
2022-05-18T03:33:51.6592137Z Running distributed/elastic/timer/local_timer_test ... [2022-05-18 03:33:51.658820]
2022-05-18T03:33:51.6592717Z Executing ['/opt/conda/bin/python', 'distributed/elastic/timer/local_timer_test.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-05-18 03:33:51.658899]
2022-05-18T03:33:52.2306058Z Test results will be stored in test-reports/python-unittest/distributed.elastic.timer.local_timer_test
2022-05-18T03:33:52.2318483Z 
2022-05-18T03:33:52.2318570Z Running tests...
2022-05-18T03:33:52.2319394Z ----------------------------------------------------------------------
2022-05-18T03:33:52.2327865Z   test_acquire_release (__main__.LocalTimerServerTest)
2022-05-18T03:33:52.5098561Z tests that: ... ok (0.278s)
2022-05-18T03:33:52.5104303Z   test_expired_timers (__main__.LocalTimerServerTest)
2022-05-18T03:33:52.5119831Z tests that a single expired timer on a process should terminate ... ok (0.002s)
2022-05-18T03:33:52.5130521Z   test_valid_timers (__main__.LocalTimerServerTest)
2022-05-18T03:33:52.5142597Z tests that valid timers are processed correctly and the process is left alone ... ok (0.002s)
2022-05-18T03:33:52.5149502Z   test_watchdog_call_count (__main__.LocalTimerServerTest)
2022-05-18T03:33:52.6169908Z checks that the watchdog function ran wait/interval +- 1 times ... ok (0.102s)
2022-05-18T03:33:52.6171624Z   test_watchdog_empty_queue (__main__.LocalTimerServerTest)
2022-05-18T03:33:52.6278870Z checks that the watchdog can run on an empty queue ... ok (0.011s)
2022-05-18T03:33:52.6334954Z   test_client_interaction (__main__.LocalTimerTest) ... ok (0.005s)
2022-05-18T03:33:52.6447228Z   test_exception_propagation (__main__.LocalTimerTest) ... ok (0.011s)
2022-05-18T03:33:52.6454860Z   test_get_timer_recursive (__main__.LocalTimerTest)
2022-05-18T03:33:53.6111019Z If a function acquires a countdown timer with default scope, ... ok (0.966s)
2022-05-18T03:33:53.7140366Z   test_happy_path (__main__.LocalTimerTest) ... ok (0.103s)
2022-05-18T03:33:53.7252324Z   test_no_client (__main__.LocalTimerTest) ... ok (0.011s)
2022-05-18T03:33:53.8492555Z   test_timer (__main__.LocalTimerTest) ... ok (0.124s)
2022-05-18T03:33:53.8718066Z   test_get (__main__.MultiprocessingRequestQueueTest) ... ok (0.022s)
2022-05-18T03:33:53.8724240Z   test_get_less_than_size (__main__.MultiprocessingRequestQueueTest)
2022-05-18T03:33:54.3776697Z Tests slow producer. ... ok (0.506s)
2022-05-18T03:33:54.3788442Z   test_get_size (__main__.MultiprocessingRequestQueueTest)
2022-05-18T03:33:55.2860033Z Creates a "producer" process that enqueues ``n`` elements ... ok (0.908s)
2022-05-18T03:33:55.2860341Z 
2022-05-18T03:33:55.2860728Z ----------------------------------------------------------------------
2022-05-18T03:33:55.2860981Z Ran 14 tests in 3.054s
2022-05-18T03:33:55.2861096Z 
2022-05-18T03:33:55.2861158Z OK
2022-05-18T03:33:55.2861236Z 
2022-05-18T03:33:55.2861322Z Generating XML reports...
2022-05-18T03:33:55.2904962Z Generated XML report: test-reports/python-unittest/distributed.elastic.timer.local_timer_test/TEST-LocalTimerServerTest-20220518033352.xml
2022-05-18T03:33:55.2911347Z Generated XML report: test-reports/python-unittest/distributed.elastic.timer.local_timer_test/TEST-LocalTimerTest-20220518033352.xml
2022-05-18T03:33:55.2916609Z Generated XML report: test-reports/python-unittest/distributed.elastic.timer.local_timer_test/TEST-MultiprocessingRequestQueueTest-20220518033352.xml
2022-05-18T03:33:55.5750101Z Running distributed/elastic/utils/distributed_test ... [2022-05-18 03:33:55.574584]
2022-05-18T03:33:55.5750726Z Executing ['/opt/conda/bin/python', 'distributed/elastic/utils/distributed_test.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-05-18 03:33:55.574681]
2022-05-18T03:33:56.1831415Z Test results will be stored in test-reports/python-unittest/distributed.elastic.utils.distributed_test
2022-05-18T03:33:56.1843149Z 
2022-05-18T03:33:56.1843616Z Running tests...
2022-05-18T03:33:56.1844018Z ----------------------------------------------------------------------
2022-05-18T03:33:56.4698875Z   test_create_store_multi (__main__.DistributedUtilTest) ... ok (0.285s)
2022-05-18T03:33:56.4709555Z   test_create_store_no_port_multi (__main__.DistributedUtilTest) ... ok (0.001s)
2022-05-18T03:33:56.4714373Z   test_create_store_single_server (__main__.DistributedUtilTest) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/66207 for allplatform(s) . If you're seeing this on your local machine and would like to enable this test, please make sure IN_CI is not set and you are not using the flag --import-disabled-tests. (0.000s)
2022-05-18T03:33:59.4857688Z   test_create_store_timeout_on_server (__main__.DistributedUtilTest) ... ok (3.014s)
2022-05-18T03:33:59.4866390Z   test_create_store_timeout_on_worker (__main__.DistributedUtilTest) ... [E socket.cpp:793] [c10d] The client socket has timed out after 1s while trying to connect to (e67028644d5a, 0).
2022-05-18T03:33:59.4868285Z ok (0.001s)
2022-05-18T03:33:59.4881860Z   test_port_already_in_use_on_server (__main__.DistributedUtilTest) ... [W socket.cpp:401] [c10d] The server socket has failed to bind to [::]:43589 (errno: 98 - Address already in use).
2022-05-18T03:33:59.4892342Z [W socket.cpp:401] [c10d] The server socket has failed to bind to 0.0.0.0:43589 (errno: 98 - Address already in use).
2022-05-18T03:33:59.4892711Z [E socket.cpp:435] [c10d] The server socket has failed to listen on any local network address.
2022-05-18T03:33:59.4894812Z ok (0.003s)
2022-05-18T03:33:59.4917930Z   test_port_already_in_use_on_worker (__main__.DistributedUtilTest) ... [E socket.cpp:793] [c10d] The client socket has timed out after 1s while trying to connect to (e67028644d5a, 51217).
2022-05-18T03:33:59.4919766Z ok (0.002s)
2022-05-18T03:33:59.4919988Z 
2022-05-18T03:33:59.4920574Z ----------------------------------------------------------------------
2022-05-18T03:33:59.4921071Z Ran 7 tests in 3.308s
2022-05-18T03:33:59.4921281Z 
2022-05-18T03:33:59.4921421Z OK (skipped=1)
2022-05-18T03:33:59.4921615Z 
2022-05-18T03:33:59.4921749Z Generating XML reports...
2022-05-18T03:33:59.4954586Z Generated XML report: test-reports/python-unittest/distributed.elastic.utils.distributed_test/TEST-DistributedUtilTest-20220518033356.xml
2022-05-18T03:33:59.6759257Z Running distributed/elastic/utils/logging_test ... [2022-05-18 03:33:59.675544]
2022-05-18T03:33:59.6759887Z Executing ['/opt/conda/bin/python', 'distributed/elastic/utils/logging_test.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-05-18 03:33:59.675622]
2022-05-18T03:34:00.2771793Z Test results will be stored in test-reports/python-unittest/distributed.elastic.utils.logging_test
2022-05-18T03:34:00.2782621Z 
2022-05-18T03:34:00.2783347Z Running tests...
2022-05-18T03:34:00.2783742Z ----------------------------------------------------------------------
2022-05-18T03:34:00.5721983Z   test_derive_module_name (__main__.LoggingTest) ... ok (0.294s)
2022-05-18T03:34:00.5741207Z   test_logger_name (__main__.LoggingTest) ... ok (0.002s)
2022-05-18T03:34:00.5741686Z 
2022-05-18T03:34:00.5742138Z ----------------------------------------------------------------------
2022-05-18T03:34:00.5742633Z Ran 2 tests in 0.296s
2022-05-18T03:34:00.5742757Z 
2022-05-18T03:34:00.5742818Z OK
2022-05-18T03:34:00.5743061Z 
2022-05-18T03:34:00.5743158Z Generating XML reports...
2022-05-18T03:34:00.5767502Z Generated XML report: test-reports/python-unittest/distributed.elastic.utils.logging_test/TEST-LoggingTest-20220518033400.xml
2022-05-18T03:34:00.7373017Z Running distributed/elastic/utils/util_test ... [2022-05-18 03:34:00.736977]
2022-05-18T03:34:00.7373564Z Executing ['/opt/conda/bin/python', 'distributed/elastic/utils/util_test.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-05-18 03:34:00.737053]
2022-05-18T03:34:01.2928566Z Test results will be stored in test-reports/python-unittest/distributed.elastic.utils.util_test
2022-05-18T03:34:01.2940279Z 
2022-05-18T03:34:01.2940373Z Running tests...
2022-05-18T03:34:01.2940834Z ----------------------------------------------------------------------
2022-05-18T03:34:01.5730389Z   test_get_all_rank_0 (__main__.StoreUtilTest) ... ok (0.279s)
2022-05-18T03:34:01.5745620Z   test_get_all_rank_n (__main__.StoreUtilTest) ... ok (0.002s)
2022-05-18T03:34:01.5765140Z   test_synchronize (__main__.StoreUtilTest) ... ok (0.002s)
2022-05-18T03:34:01.6307171Z   test_get_logger (__main__.UtilTest) ... ok (0.054s)
2022-05-18T03:34:01.6313809Z   test_get_logger_custom_name (__main__.UtilTest) ... ok (0.001s)
2022-05-18T03:34:01.6322321Z   test_get_logger_different (__main__.UtilTest) ... ok (0.001s)
2022-05-18T03:34:01.6333318Z   test_get_logger_none (__main__.UtilTest) ... ok (0.001s)
2022-05-18T03:34:01.6333513Z 
2022-05-18T03:34:01.6333992Z ----------------------------------------------------------------------
2022-05-18T03:34:01.6334468Z Ran 7 tests in 0.339s
2022-05-18T03:34:01.6334652Z 
2022-05-18T03:34:01.6334715Z OK
2022-05-18T03:34:01.6334808Z 
2022-05-18T03:34:01.6334900Z Generating XML reports...
2022-05-18T03:34:01.6361195Z Generated XML report: test-reports/python-unittest/distributed.elastic.utils.util_test/TEST-StoreUtilTest-20220518033401.xml
2022-05-18T03:34:01.6365614Z Generated XML report: test-reports/python-unittest/distributed.elastic.utils.util_test/TEST-UtilTest-20220518033401.xml
2022-05-18T03:34:01.7898215Z Running distributed/fsdp/test_distributed_checkpoint ... [2022-05-18 03:34:01.789459]
2022-05-18T03:34:01.7898851Z Executing ['/opt/conda/bin/python', 'distributed/fsdp/test_distributed_checkpoint.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-05-18 03:34:01.789537]
2022-05-18T03:34:02.3674912Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_distributed_checkpoint
2022-05-18T03:34:02.3685650Z 
2022-05-18T03:34:02.3685981Z Running tests...
2022-05-18T03:34:02.3686585Z ----------------------------------------------------------------------
2022-05-18T03:34:02.6480962Z   test_distributed_checkpoint_state_dict_type_StateDictType_LOCAL_STATE_DICT (__main__.TestDistributedCheckpoint) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1341
2022-05-18T03:34:02.6502550Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1342
2022-05-18T03:34:03.2223945Z dist init r=1, world=2
2022-05-18T03:34:03.2224318Z dist init r=0, world=2
2022-05-18T03:34:03.2432085Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:34:03.2432659Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:34:03.2433371Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:34:03.2433885Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:34:03.2437354Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:34:03.2437906Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:34:03.4527325Z skip: Need at least 2 CUDA devices (1.084s)
2022-05-18T03:34:03.4576540Z   test_distributed_checkpoint_state_dict_type_StateDictType_SHARDED_STATE_DICT (__main__.TestDistributedCheckpoint) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1369
2022-05-18T03:34:03.4601299Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1370
2022-05-18T03:34:04.0330096Z dist init r=0, world=2
2022-05-18T03:34:04.0338872Z dist init r=1, world=2
2022-05-18T03:34:04.0446997Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:34:04.0447557Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:34:04.0448331Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:34:04.0448866Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:34:04.0552389Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:34:04.0553027Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:34:04.2623228Z skip: Need at least 2 CUDA devices (0.809s)
2022-05-18T03:34:04.2623423Z 
2022-05-18T03:34:04.2623737Z ----------------------------------------------------------------------
2022-05-18T03:34:04.2623996Z Ran 2 tests in 1.894s
2022-05-18T03:34:04.2624115Z 
2022-05-18T03:34:04.2624186Z OK (skipped=2)
2022-05-18T03:34:04.2624296Z 
2022-05-18T03:34:04.2624380Z Generating XML reports...
2022-05-18T03:34:04.2660896Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_distributed_checkpoint/TEST-TestDistributedCheckpoint-20220518033402.xml
2022-05-18T03:34:04.4554645Z Running distributed/fsdp/test_flatten_params_wrapper ... [2022-05-18 03:34:04.455085]
2022-05-18T03:34:04.4555358Z Executing ['/opt/conda/bin/python', 'distributed/fsdp/test_flatten_params_wrapper.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-05-18 03:34:04.455167]
2022-05-18T03:34:05.0241497Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_flatten_params_wrapper
2022-05-18T03:34:05.0254370Z 
2022-05-18T03:34:05.0254449Z Running tests...
2022-05-18T03:34:05.0255008Z ----------------------------------------------------------------------
2022-05-18T03:34:05.2978951Z   test_empty_module (__main__.TestFlattenParams) ... ok (0.272s)
2022-05-18T03:34:05.3051907Z   test_flatten_nothing (__main__.TestFlattenParams) ... ok (0.007s)
2022-05-18T03:34:05.3128404Z   test_num_params (__main__.TestFlattenParams) ... ok (0.008s)
2022-05-18T03:34:05.3327568Z   test_output (__main__.TestFlattenParams) ... ok (0.020s)
2022-05-18T03:34:05.3422673Z   test_partial_flattening (__main__.TestFlattenParams) ... ok (0.010s)
2022-05-18T03:34:05.3509405Z   test_sharded_flat_param (__main__.TestFlattenParams) ... ok (0.009s)
2022-05-18T03:34:05.3584850Z   test_shared_params_num_params (__main__.TestFlattenParams) ... ok (0.008s)
2022-05-18T03:34:05.3760679Z   test_shared_params_output (__main__.TestFlattenParams) ... ok (0.017s)
2022-05-18T03:34:05.4077522Z   test_shared_params_pnorm_after_step (__main__.TestFlattenParams) ... ok (0.032s)
2022-05-18T03:34:05.4082780Z   test_empty_module (__main__.TestFlattenParamsCUDA) ... skip: test requires a GPU (0.001s)
2022-05-18T03:34:05.4086464Z   test_flatten_nothing (__main__.TestFlattenParamsCUDA) ... skip: test requires a GPU (0.000s)
2022-05-18T03:34:05.4088903Z   test_num_params (__main__.TestFlattenParamsCUDA) ... skip: test requires a GPU (0.000s)
2022-05-18T03:34:05.4091355Z   test_output (__main__.TestFlattenParamsCUDA) ... skip: test requires a GPU (0.000s)
2022-05-18T03:34:05.4106368Z   test_partial_flattening (__main__.TestFlattenParamsCUDA) ... skip: test requires a GPU (0.001s)
2022-05-18T03:34:05.4146036Z   test_sharded_flat_param (__main__.TestFlattenParamsCUDA) ... skip: test requires a GPU (0.004s)
2022-05-18T03:34:05.4148677Z   test_shared_params_num_params (__main__.TestFlattenParamsCUDA) ... skip: test requires a GPU (0.000s)
2022-05-18T03:34:05.4151506Z   test_shared_params_output (__main__.TestFlattenParamsCUDA) ... skip: test requires a GPU (0.000s)
2022-05-18T03:34:05.4155959Z   test_shared_params_pnorm_after_step (__main__.TestFlattenParamsCUDA) ... skip: test requires a GPU (0.000s)
2022-05-18T03:34:05.4161007Z   test_empty_module (__main__.TestFlattenParamsCUDAHalf) ... skip: test requires a GPU (0.000s)
2022-05-18T03:34:05.4163966Z   test_flatten_nothing (__main__.TestFlattenParamsCUDAHalf) ... skip: test requires a GPU (0.000s)
2022-05-18T03:34:05.4166138Z   test_num_params (__main__.TestFlattenParamsCUDAHalf) ... skip: test requires a GPU (0.000s)
2022-05-18T03:34:05.4168473Z   test_output (__main__.TestFlattenParamsCUDAHalf) ... skip: test requires a GPU (0.000s)
2022-05-18T03:34:05.4183232Z   test_partial_flattening (__main__.TestFlattenParamsCUDAHalf) ... skip: test requires a GPU (0.001s)
2022-05-18T03:34:05.4223056Z   test_sharded_flat_param (__main__.TestFlattenParamsCUDAHalf) ... skip: test requires a GPU (0.004s)
2022-05-18T03:34:05.4225176Z   test_shared_params_num_params (__main__.TestFlattenParamsCUDAHalf) ... skip: test requires a GPU (0.000s)
2022-05-18T03:34:05.4227431Z   test_shared_params_output (__main__.TestFlattenParamsCUDAHalf) ... skip: test requires a GPU (0.000s)
2022-05-18T03:34:05.4232376Z   test_shared_params_pnorm_after_step (__main__.TestFlattenParamsCUDAHalf) ... skip: test requires a GPU (0.000s)
2022-05-18T03:34:05.4232727Z 
2022-05-18T03:34:05.4233048Z ----------------------------------------------------------------------
2022-05-18T03:34:05.4233299Z Ran 27 tests in 0.398s
2022-05-18T03:34:05.4233415Z 
2022-05-18T03:34:05.4233488Z OK (skipped=18)
2022-05-18T03:34:05.4233594Z 
2022-05-18T03:34:05.4233666Z Generating XML reports...
2022-05-18T03:34:05.4263794Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_flatten_params_wrapper/TEST-TestFlattenParams-20220518033405.xml
2022-05-18T03:34:05.4285925Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_flatten_params_wrapper/TEST-TestFlattenParamsCUDA-20220518033405.xml
2022-05-18T03:34:05.4286615Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_flatten_params_wrapper/TEST-TestFlattenParamsCUDAHalf-20220518033405.xml
2022-05-18T03:34:05.5941132Z Running distributed/fsdp/test_fsdp_apply ... [2022-05-18 03:34:05.593708]
2022-05-18T03:34:05.5941673Z Executing ['/opt/conda/bin/python', 'distributed/fsdp/test_fsdp_apply.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-05-18 03:34:05.593794]
2022-05-18T03:34:06.1652070Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_apply
2022-05-18T03:34:06.1668295Z 
2022-05-18T03:34:06.1668441Z Running tests...
2022-05-18T03:34:06.1669035Z ----------------------------------------------------------------------
2022-05-18T03:34:06.1674786Z   test_apply_in_summon_raises_error (__main__.TestApply)
2022-05-18T03:34:06.4462643Z Ensures that if user calls apply() on FSDP instance within full param ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1421
2022-05-18T03:34:06.4483865Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1422
2022-05-18T03:34:07.0187731Z dist init r=0, world=2
2022-05-18T03:34:07.0201387Z dist init r=1, world=2
2022-05-18T03:34:07.0309715Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:34:07.0310336Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:34:07.0311051Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:34:07.0311579Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:34:07.0414813Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:34:07.0415300Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:34:07.2508879Z skip: Need at least 2 CUDA devices (1.084s)
2022-05-18T03:34:07.2514294Z   test_nested_module_apply (__main__.TestApply)
2022-05-18T03:34:07.2549574Z Checks apply() modifies weights appropriately on a nested FSDP instance. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1449
2022-05-18T03:34:07.2575898Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1450
2022-05-18T03:34:07.8293246Z dist init r=0, world=2
2022-05-18T03:34:07.8309534Z dist init r=1, world=2
2022-05-18T03:34:07.8417566Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:34:07.8418310Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:34:07.8418963Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:34:07.8419492Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:34:07.8523981Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:34:07.8524559Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:34:08.0596934Z skip: Need at least 2 CUDA devices (0.809s)
2022-05-18T03:34:08.0601284Z   test_transformer_module_apply (__main__.TestApply)
2022-05-18T03:34:08.0636546Z Checks apply() modifies weights appropriately on a wrapped Transformer ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1477
2022-05-18T03:34:08.0661421Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1478
2022-05-18T03:34:08.6429304Z dist init r=1, world=2
2022-05-18T03:34:08.6501297Z dist init r=0, world=2
2022-05-18T03:34:08.6637174Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:34:08.6637628Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:34:08.6638243Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:34:08.6639026Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:34:08.6642490Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:34:08.6642859Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:34:08.8683009Z skip: Need at least 2 CUDA devices (0.808s)
2022-05-18T03:34:08.8683223Z 
2022-05-18T03:34:08.8683680Z ----------------------------------------------------------------------
2022-05-18T03:34:08.8683953Z Ran 3 tests in 2.701s
2022-05-18T03:34:08.8684070Z 
2022-05-18T03:34:08.8684130Z OK (skipped=3)
2022-05-18T03:34:08.8684247Z 
2022-05-18T03:34:08.8684368Z Generating XML reports...
2022-05-18T03:34:08.8722073Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_apply/TEST-TestApply-20220518033406.xml
2022-05-18T03:34:09.0671786Z Running distributed/fsdp/test_fsdp_checkpoint ... [2022-05-18 03:34:09.066775]
2022-05-18T03:34:09.0672318Z Executing ['/opt/conda/bin/python', 'distributed/fsdp/test_fsdp_checkpoint.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-05-18 03:34:09.066878]
2022-05-18T03:34:09.6404008Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_checkpoint
2022-05-18T03:34:09.6414715Z 
2022-05-18T03:34:09.6414811Z Running tests...
2022-05-18T03:34:09.6417019Z ----------------------------------------------------------------------
2022-05-18T03:34:09.9215848Z   test_basic_checkpoint_end_to_end_cpu_offload_CPUOffload(offload_params=False)_offload_activations_False (__main__.TestFSDPCheckpoint) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1516
2022-05-18T03:34:09.9236834Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1517
2022-05-18T03:34:09.9259264Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1518
2022-05-18T03:34:09.9282505Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1519
2022-05-18T03:34:10.5572428Z dist init r=2, world=4
2022-05-18T03:34:10.5869254Z dist init r=0, world=4
2022-05-18T03:34:10.6037073Z dist init r=1, world=4
2022-05-18T03:34:10.6202450Z dist init r=3, world=4
2022-05-18T03:34:10.6347224Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:34:10.6447875Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:34:10.6549439Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:34:10.6549928Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:34:10.6550825Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:10.6551717Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:10.6552496Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:10.6553403Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:10.6657143Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:34:10.6657740Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:34:10.6658296Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:34:10.6658830Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:34:10.8311809Z skip: Need at least 2 CUDA devices (1.189s)
2022-05-18T03:34:10.8334101Z   test_basic_checkpoint_end_to_end_cpu_offload_CPUOffload(offload_params=False)_offload_activations_True (__main__.TestFSDPCheckpoint) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/71418 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure IN_CI is not set and you are not using the flag --import-disabled-tests. (0.002s)
2022-05-18T03:34:10.8383400Z   test_basic_checkpoint_end_to_end_cpu_offload_CPUOffload(offload_params=True)_offload_activations_False (__main__.TestFSDPCheckpoint) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1572
2022-05-18T03:34:10.8408542Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1573
2022-05-18T03:34:10.8431404Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1574
2022-05-18T03:34:10.8454560Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1575
2022-05-18T03:34:11.4157387Z dist init r=2, world=4
2022-05-18T03:34:11.4693434Z dist init r=0, world=4
2022-05-18T03:34:11.4707796Z dist init r=1, world=4
2022-05-18T03:34:11.4746997Z dist init r=3, world=4
2022-05-18T03:34:11.4969128Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:34:11.5070588Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:34:11.5071115Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:34:11.5071896Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:34:11.5072736Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:11.5073274Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:11.5073955Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:11.5074477Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:11.5179408Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:34:11.5179975Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:34:11.5180522Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:34:11.5181073Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:34:11.7481060Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:34:11.7501849Z   test_basic_checkpoint_end_to_end_cpu_offload_CPUOffload(offload_params=True)_offload_activations_True (__main__.TestFSDPCheckpoint) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/70368 for platform(s) win, linux. If you're seeing this on your local machine and would like to enable this test, please make sure IN_CI is not set and you are not using the flag --import-disabled-tests. (0.002s)
2022-05-18T03:34:11.7549783Z   test_checkpoint_fsdp_wrapping_cpu_offload_CPUOffload(offload_params=False)_offload_activations_False (__main__.TestFSDPCheckpoint) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1628
2022-05-18T03:34:11.7574655Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1629
2022-05-18T03:34:11.7597455Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1630
2022-05-18T03:34:11.7620844Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1631
2022-05-18T03:34:12.3653839Z dist init r=3, world=4
2022-05-18T03:34:12.3822367Z dist init r=0, world=4
2022-05-18T03:34:12.3992768Z dist init r=1, world=4
2022-05-18T03:34:12.4153921Z dist init r=2, world=4
2022-05-18T03:34:12.4264772Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:34:12.4404185Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:34:12.4505724Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:34:12.4506459Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:34:12.4507210Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:12.4507749Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:12.4508258Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:12.4569271Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:12.4613081Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:34:12.4613649Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:34:12.4614190Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:34:12.4614917Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:34:12.6647440Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:34:12.6666551Z   test_checkpoint_fsdp_wrapping_cpu_offload_CPUOffload(offload_params=False)_offload_activations_True (__main__.TestFSDPCheckpoint) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/71009 for allplatform(s) . If you're seeing this on your local machine and would like to enable this test, please make sure IN_CI is not set and you are not using the flag --import-disabled-tests. (0.002s)
2022-05-18T03:34:12.6713960Z   test_checkpoint_fsdp_wrapping_cpu_offload_CPUOffload(offload_params=True)_offload_activations_False (__main__.TestFSDPCheckpoint) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1684
2022-05-18T03:34:12.6738766Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1685
2022-05-18T03:34:12.6761664Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1686
2022-05-18T03:34:12.6785139Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1687
2022-05-18T03:34:13.2806412Z dist init r=1, world=4
2022-05-18T03:34:13.2861945Z dist init r=0, world=4
2022-05-18T03:34:13.3021202Z dist init r=3, world=4
2022-05-18T03:34:13.3162360Z dist init r=2, world=4
2022-05-18T03:34:13.3330272Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:34:13.3431211Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:34:13.3532597Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:34:13.3533197Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:34:13.3534197Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:13.3535000Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:13.3535570Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:13.3536195Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:13.3640644Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:34:13.3641211Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:34:13.3641778Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:34:13.3642316Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:34:13.5810354Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:34:13.5830284Z   test_checkpoint_fsdp_wrapping_cpu_offload_CPUOffload(offload_params=True)_offload_activations_True (__main__.TestFSDPCheckpoint) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/71349 for allplatform(s) . If you're seeing this on your local machine and would like to enable this test, please make sure IN_CI is not set and you are not using the flag --import-disabled-tests. (0.002s)
2022-05-18T03:34:13.5831204Z 
2022-05-18T03:34:13.5831558Z ----------------------------------------------------------------------
2022-05-18T03:34:13.5831965Z Ran 8 tests in 3.941s
2022-05-18T03:34:13.5832155Z 
2022-05-18T03:34:13.5832275Z OK (skipped=8)
2022-05-18T03:34:13.5832457Z 
2022-05-18T03:34:13.5832585Z Generating XML reports...
2022-05-18T03:34:13.5871625Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_checkpoint/TEST-TestFSDPCheckpoint-20220518033409.xml
2022-05-18T03:34:13.7737107Z Running distributed/fsdp/test_fsdp_clip_grad_norm ... [2022-05-18 03:34:13.773286]
2022-05-18T03:34:13.7737970Z Executing ['/opt/conda/bin/python', 'distributed/fsdp/test_fsdp_clip_grad_norm.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-05-18 03:34:13.773365]
2022-05-18T03:34:14.3486075Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_clip_grad_norm
2022-05-18T03:34:14.3498797Z 
2022-05-18T03:34:14.3498934Z Running tests...
2022-05-18T03:34:14.3499382Z ----------------------------------------------------------------------
2022-05-18T03:34:14.3506737Z   test_fsdp_calc_grad_norm_error_norm_type_1_3 (__main__.TestCalcuGradNorm)
2022-05-18T03:34:14.6313201Z Test the abnormal cases of grad norm cal API. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1751
2022-05-18T03:34:14.6335166Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1752
2022-05-18T03:34:14.6357823Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1753
2022-05-18T03:34:14.6382074Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1754
2022-05-18T03:34:15.2685533Z dist init r=3, world=4
2022-05-18T03:34:15.2717788Z dist init r=1, world=4
2022-05-18T03:34:15.2771753Z dist init r=2, world=4
2022-05-18T03:34:15.2808659Z dist init r=0, world=4
2022-05-18T03:34:15.3082595Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:34:15.3183289Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:34:15.3285568Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:34:15.3286175Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:34:15.3286976Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:15.3287506Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:15.3288034Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:15.3288550Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:15.3393973Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:34:15.3394628Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:34:15.3395005Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:34:15.3395443Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:34:15.5411614Z skip: Need at least 2 CUDA devices (1.191s)
2022-05-18T03:34:15.5418370Z   test_fsdp_calc_grad_norm_error_norm_type_2_5 (__main__.TestCalcuGradNorm)
2022-05-18T03:34:15.5454215Z Test the abnormal cases of grad norm cal API. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1807
2022-05-18T03:34:15.5480840Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1808
2022-05-18T03:34:15.5504180Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1809
2022-05-18T03:34:15.5528341Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1810
2022-05-18T03:34:16.1484347Z dist init r=2, world=4
2022-05-18T03:34:16.1581109Z dist init r=1, world=4
2022-05-18T03:34:16.1647689Z dist init r=3, world=4
2022-05-18T03:34:16.1928428Z dist init r=0, world=4
2022-05-18T03:34:16.2195224Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:34:16.2294083Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:34:16.2295125Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:34:16.2295862Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:34:16.2296563Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:16.2297203Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:16.2298033Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:16.2298561Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:16.2403474Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:34:16.2404049Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:34:16.2404615Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:34:16.2405151Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:34:16.4554640Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:34:16.4562344Z   test_fsdp_calc_grad_norm_norm_type_2_0_nested_fsdp_False (__main__.TestCalcuGradNorm)
2022-05-18T03:34:16.4607585Z Test grad norm cal API. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1863
2022-05-18T03:34:16.4633049Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1864
2022-05-18T03:34:16.4655812Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1865
2022-05-18T03:34:16.4678705Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1866
2022-05-18T03:34:17.1487565Z dist init r=1, world=4
2022-05-18T03:34:17.1651710Z dist init r=2, world=4
2022-05-18T03:34:17.1660432Z dist init r=3, world=4
2022-05-18T03:34:17.1770911Z dist init r=0, world=4
2022-05-18T03:34:17.2072785Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:34:17.2276264Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:34:17.2276752Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:34:17.2277302Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:34:17.2278126Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:17.2278664Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:17.2279188Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:17.2279715Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:17.2383828Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:34:17.2384392Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:34:17.2384894Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:34:17.2385256Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:34:17.4707087Z skip: Need at least 2 CUDA devices (1.015s)
2022-05-18T03:34:17.4714226Z   test_fsdp_calc_grad_norm_norm_type_2_0_nested_fsdp_True (__main__.TestCalcuGradNorm)
2022-05-18T03:34:17.4750089Z Test grad norm cal API. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1919
2022-05-18T03:34:17.4776381Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1920
2022-05-18T03:34:17.4799588Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1921
2022-05-18T03:34:17.4822762Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1922
2022-05-18T03:34:18.0869282Z dist init r=1, world=4
2022-05-18T03:34:18.0992651Z dist init r=3, world=4
2022-05-18T03:34:18.0995319Z dist init r=2, world=4
2022-05-18T03:34:18.1045650Z dist init r=0, world=4
2022-05-18T03:34:18.1201360Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:34:18.1404008Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:34:18.1505749Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:34:18.1506600Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:18.1507078Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:34:18.1507574Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:18.1508083Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:18.1508597Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:18.1613847Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:34:18.1614506Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:34:18.1614972Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:34:18.1615462Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:34:18.3849390Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:34:18.3857115Z   test_fsdp_calc_grad_norm_norm_type_inf_nested_fsdp_False (__main__.TestCalcuGradNorm)
2022-05-18T03:34:18.3893338Z Test grad norm cal API. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1975
2022-05-18T03:34:18.3918891Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1976
2022-05-18T03:34:18.3941585Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1977
2022-05-18T03:34:18.3965576Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1978
2022-05-18T03:34:18.9768619Z dist init r=3, world=4
2022-05-18T03:34:19.0158260Z dist init r=1, world=4
2022-05-18T03:34:19.0249830Z dist init r=2, world=4
2022-05-18T03:34:19.0309815Z dist init r=0, world=4
2022-05-18T03:34:19.0568478Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:34:19.0669753Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:34:19.0670454Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:34:19.0670918Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:34:19.0671722Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:19.0672405Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:19.0672928Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:19.0673452Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:19.0778754Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:34:19.0779727Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:34:19.0780550Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:34:19.0781368Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:34:19.2992663Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:34:19.2999980Z   test_fsdp_calc_grad_norm_norm_type_inf_nested_fsdp_True (__main__.TestCalcuGradNorm)
2022-05-18T03:34:19.3035825Z Test grad norm cal API. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2031
2022-05-18T03:34:19.3061418Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2032
2022-05-18T03:34:19.3085345Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2033
2022-05-18T03:34:19.3109421Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2034
2022-05-18T03:34:19.8871779Z dist init r=2, world=4
2022-05-18T03:34:19.8900066Z dist init r=3, world=4
2022-05-18T03:34:19.9502682Z dist init r=0, world=4
2022-05-18T03:34:19.9574045Z dist init r=1, world=4
2022-05-18T03:34:19.9885316Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:34:19.9986336Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:34:20.0089320Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:34:20.0090754Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:20.0091625Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:20.0092128Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:34:20.0092653Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:20.0093269Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:20.0099096Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:34:20.0099599Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:34:20.0100118Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:34:20.0100646Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:34:20.2135944Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:34:20.2142345Z   test_fsdp_clip_grad_norm_norm_type_2_0_nested_fsdp_False_cpu_offload_CPUOffload(offload_params=False) (__main__.TestClipGradNorm)
2022-05-18T03:34:20.2178194Z Test FSDP with clip grad norm. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2087
2022-05-18T03:34:20.2204458Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2088
2022-05-18T03:34:20.2227793Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2089
2022-05-18T03:34:20.2251903Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2090
2022-05-18T03:34:20.8074649Z dist init r=0, world=4
2022-05-18T03:34:20.8369451Z dist init r=1, world=4
2022-05-18T03:34:20.8597862Z dist init r=2, world=4
2022-05-18T03:34:20.8632977Z dist init r=3, world=4
2022-05-18T03:34:20.8880820Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:34:20.8881413Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:34:20.8982601Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:34:20.8983594Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:34:20.8984830Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:20.8985466Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:20.8985991Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:20.8986492Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:20.9091474Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:34:20.9092164Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:34:20.9092821Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:34:20.9093458Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:34:21.1279621Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:34:21.1284676Z   test_fsdp_clip_grad_norm_norm_type_2_0_nested_fsdp_False_cpu_offload_CPUOffload(offload_params=True) (__main__.TestClipGradNorm)
2022-05-18T03:34:21.1319850Z Test FSDP with clip grad norm. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2143
2022-05-18T03:34:21.1345057Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2144
2022-05-18T03:34:21.1367949Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2145
2022-05-18T03:34:21.1391759Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2146
2022-05-18T03:34:21.7384435Z dist init r=0, world=4
2022-05-18T03:34:21.7446022Z dist init r=1, world=4
2022-05-18T03:34:21.7479940Z dist init r=2, world=4
2022-05-18T03:34:21.7836753Z dist init r=3, world=4
2022-05-18T03:34:21.7944722Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:34:21.7991369Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:34:21.8093076Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:34:21.8093578Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:34:21.8094213Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:21.8094759Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:21.8095275Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:21.8148430Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:21.8202192Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:34:21.8202849Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:34:21.8203501Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:34:21.8204127Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:34:22.0418305Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:34:22.0424546Z   test_fsdp_clip_grad_norm_norm_type_2_0_nested_fsdp_True_cpu_offload_CPUOffload(offload_params=False) (__main__.TestClipGradNorm)
2022-05-18T03:34:22.0460829Z Test FSDP with clip grad norm. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2199
2022-05-18T03:34:22.0486823Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2200
2022-05-18T03:34:22.0509994Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2201
2022-05-18T03:34:22.0533899Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2202
2022-05-18T03:34:22.6273055Z dist init r=3, world=4
2022-05-18T03:34:22.6331506Z dist init r=0, world=4
2022-05-18T03:34:22.6561528Z dist init r=1, world=4
2022-05-18T03:34:22.6705102Z dist init r=2, world=4
2022-05-18T03:34:22.6943841Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:34:22.6944260Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:34:22.7046114Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:34:22.7047032Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:22.7047442Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:34:22.7047943Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:22.7048455Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:22.7048960Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:22.7153913Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:34:22.7154469Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:34:22.7155029Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:34:22.7155578Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:34:22.8558673Z skip: Need at least 2 CUDA devices (0.814s)
2022-05-18T03:34:22.8564169Z   test_fsdp_clip_grad_norm_norm_type_2_0_nested_fsdp_True_cpu_offload_CPUOffload(offload_params=True) (__main__.TestClipGradNorm)
2022-05-18T03:34:22.8598784Z Test FSDP with clip grad norm. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2255
2022-05-18T03:34:22.8624869Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2256
2022-05-18T03:34:22.8647237Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2257
2022-05-18T03:34:22.8671133Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2258
2022-05-18T03:34:23.5074374Z dist init r=1, world=4
2022-05-18T03:34:23.5217418Z dist init r=3, world=4
2022-05-18T03:34:23.5246011Z dist init r=2, world=4
2022-05-18T03:34:23.5280659Z dist init r=0, world=4
2022-05-18T03:34:23.5583648Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:34:23.5685637Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:34:23.5687794Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:34:23.5688455Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:23.5688864Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:34:23.5689353Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:23.5690042Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:23.5690625Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:23.5793413Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:34:23.5794126Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:34:23.5794805Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:34:23.5795343Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:34:23.7696936Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:34:23.7703191Z   test_fsdp_clip_grad_norm_norm_type_inf_nested_fsdp_False_cpu_offload_CPUOffload(offload_params=False) (__main__.TestClipGradNorm)
2022-05-18T03:34:23.7741019Z Test FSDP with clip grad norm. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2311
2022-05-18T03:34:23.7767065Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2312
2022-05-18T03:34:23.7789807Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2313
2022-05-18T03:34:23.7813461Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2314
2022-05-18T03:34:24.3911765Z dist init r=3, world=4
2022-05-18T03:34:24.4015125Z dist init r=0, world=4
2022-05-18T03:34:24.4050487Z dist init r=2, world=4
2022-05-18T03:34:24.4118574Z dist init r=1, world=4
2022-05-18T03:34:24.4361202Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:34:24.4461582Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:34:24.4563790Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:34:24.4564406Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:34:24.4565383Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:24.4566267Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:24.4566883Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:24.4569810Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:24.4572206Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:34:24.4572754Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:34:24.4573284Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:34:24.4573826Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:34:24.6840079Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:34:24.6846811Z   test_fsdp_clip_grad_norm_norm_type_inf_nested_fsdp_False_cpu_offload_CPUOffload(offload_params=True) (__main__.TestClipGradNorm)
2022-05-18T03:34:24.6883104Z Test FSDP with clip grad norm. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2367
2022-05-18T03:34:24.6909482Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2368
2022-05-18T03:34:24.6932938Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2369
2022-05-18T03:34:24.6957095Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2370
2022-05-18T03:34:25.3021456Z dist init r=3, world=4
2022-05-18T03:34:25.3287139Z dist init r=0, world=4
2022-05-18T03:34:25.3302539Z dist init r=1, world=4
2022-05-18T03:34:25.3368494Z dist init r=2, world=4
2022-05-18T03:34:25.3697913Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:34:25.3698331Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:34:25.3799052Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:34:25.3799457Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:34:25.3800285Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:25.3800945Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:25.3801679Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:25.3802497Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:25.3907713Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:34:25.3908371Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:34:25.3908911Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:34:25.3909479Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:34:25.5983594Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:34:25.5990011Z   test_fsdp_clip_grad_norm_norm_type_inf_nested_fsdp_True_cpu_offload_CPUOffload(offload_params=False) (__main__.TestClipGradNorm)
2022-05-18T03:34:25.6026141Z Test FSDP with clip grad norm. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2423
2022-05-18T03:34:25.6052375Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2424
2022-05-18T03:34:25.6075516Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2425
2022-05-18T03:34:25.6099479Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2426
2022-05-18T03:34:26.1797192Z dist init r=1, world=4
2022-05-18T03:34:26.2075022Z dist init r=0, world=4
2022-05-18T03:34:26.2352240Z dist init r=2, world=4
2022-05-18T03:34:26.2641111Z dist init r=3, world=4
2022-05-18T03:34:26.2748866Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:34:26.2787307Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:34:26.2888245Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:34:26.2888735Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:34:26.2889890Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:26.2890428Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:26.2890949Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:26.2952699Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:26.2996936Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:34:26.2997479Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:34:26.2997856Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:34:26.2998390Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:34:26.5126242Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:34:26.5131672Z   test_fsdp_clip_grad_norm_norm_type_inf_nested_fsdp_True_cpu_offload_CPUOffload(offload_params=True) (__main__.TestClipGradNorm)
2022-05-18T03:34:26.5166953Z Test FSDP with clip grad norm. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2479
2022-05-18T03:34:26.5192214Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2480
2022-05-18T03:34:26.5215645Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2481
2022-05-18T03:34:26.5239857Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2482
2022-05-18T03:34:27.1148220Z dist init r=3, world=4
2022-05-18T03:34:27.1169408Z dist init r=2, world=4
2022-05-18T03:34:27.1478407Z dist init r=0, world=4
2022-05-18T03:34:27.1652023Z dist init r=1, world=4
2022-05-18T03:34:27.1859463Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:34:27.1861031Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:34:27.1963063Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:34:27.1963936Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:34:27.1964994Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:27.1965690Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:27.1966625Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:27.2065255Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:27.2073479Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:34:27.2074169Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:34:27.2074835Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:34:27.2075337Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:34:27.4267428Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:34:27.4267693Z 
2022-05-18T03:34:27.4268234Z ----------------------------------------------------------------------
2022-05-18T03:34:27.4268574Z Ran 14 tests in 13.077s
2022-05-18T03:34:27.4268691Z 
2022-05-18T03:34:27.4268765Z OK (skipped=14)
2022-05-18T03:34:27.4268873Z 
2022-05-18T03:34:27.4268974Z Generating XML reports...
2022-05-18T03:34:27.4306537Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_clip_grad_norm/TEST-TestCalcuGradNorm-20220518033414.xml
2022-05-18T03:34:27.4317930Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_clip_grad_norm/TEST-TestClipGradNorm-20220518033414.xml
2022-05-18T03:34:27.6200461Z Running distributed/fsdp/test_fsdp_comm ... [2022-05-18 03:34:27.619634]
2022-05-18T03:34:27.6201069Z Executing ['/opt/conda/bin/python', 'distributed/fsdp/test_fsdp_comm.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-05-18 03:34:27.619714]
2022-05-18T03:34:28.1929460Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_comm
2022-05-18T03:34:28.1940970Z 
2022-05-18T03:34:28.1941066Z Running tests...
2022-05-18T03:34:28.1941518Z ----------------------------------------------------------------------
2022-05-18T03:34:28.1966351Z   test_communication_nested_model_False_use_no_sync_False_sharding_strategy_None (__main__.TestCommunication)
2022-05-18T03:34:28.4761995Z Tests FSDP's communication cost in terms of calls to collective ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2546
2022-05-18T03:34:28.4782860Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2547
2022-05-18T03:34:28.4804481Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2548
2022-05-18T03:34:28.4827697Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2549
2022-05-18T03:34:29.1487524Z dist init r=1, world=4
2022-05-18T03:34:29.1702616Z dist init r=0, world=4
2022-05-18T03:34:29.1703185Z dist init r=3, world=4
2022-05-18T03:34:29.2027070Z dist init r=2, world=4
2022-05-18T03:34:29.2213718Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:34:29.2314080Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:34:29.2315286Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:29.2315912Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:34:29.2316376Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:34:29.2317130Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:29.2317668Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:29.2318188Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:29.2421092Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:34:29.2421773Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:34:29.2422331Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:34:29.2422863Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:34:29.3857742Z skip: Need at least 2 CUDA devices (1.191s)
2022-05-18T03:34:29.3879093Z   test_communication_nested_model_False_use_no_sync_False_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestCommunication)
2022-05-18T03:34:29.3913049Z Tests FSDP's communication cost in terms of calls to collective ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2602
2022-05-18T03:34:29.3937900Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2603
2022-05-18T03:34:29.3960858Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2604
2022-05-18T03:34:29.3984518Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2605
2022-05-18T03:34:30.0137489Z dist init r=1, world=4
2022-05-18T03:34:30.0240940Z dist init r=3, world=4
2022-05-18T03:34:30.0263919Z dist init r=2, world=4
2022-05-18T03:34:30.0363624Z dist init r=0, world=4
2022-05-18T03:34:30.0675681Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:34:30.0776718Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:34:30.0878365Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:34:30.0878985Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:34:30.0880003Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:30.0880741Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:30.0881447Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:30.0882134Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:30.0886618Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:34:30.0887203Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:34:30.0887581Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:34:30.0888044Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:34:30.3010898Z skip: Need at least 2 CUDA devices (0.915s)
2022-05-18T03:34:30.3029079Z   test_communication_nested_model_False_use_no_sync_True_sharding_strategy_None (__main__.TestCommunication)
2022-05-18T03:34:30.3064608Z Tests FSDP's communication cost in terms of calls to collective ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2658
2022-05-18T03:34:30.3090445Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2659
2022-05-18T03:34:30.3114112Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2660
2022-05-18T03:34:30.3137540Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2661
2022-05-18T03:34:30.9014058Z dist init r=0, world=4
2022-05-18T03:34:30.9150920Z dist init r=3, world=4
2022-05-18T03:34:30.9591195Z dist init r=2, world=4
2022-05-18T03:34:30.9591433Z dist init r=1, world=4
2022-05-18T03:34:30.9801800Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:34:30.9862820Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:34:30.9963474Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:34:30.9964366Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:34:30.9965020Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:30.9965544Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:30.9966071Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:31.0004942Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:31.0072193Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:34:31.0072764Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:34:31.0073309Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:34:31.0073866Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:34:31.2164257Z skip: Need at least 2 CUDA devices (0.915s)
2022-05-18T03:34:31.2183589Z   test_communication_nested_model_False_use_no_sync_True_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestCommunication)
2022-05-18T03:34:31.2220150Z Tests FSDP's communication cost in terms of calls to collective ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2714
2022-05-18T03:34:31.2245488Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2715
2022-05-18T03:34:31.2268811Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2716
2022-05-18T03:34:31.2292692Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2717
2022-05-18T03:34:31.8018299Z dist init r=2, world=4
2022-05-18T03:34:31.8277270Z dist init r=3, world=4
2022-05-18T03:34:31.8511038Z dist init r=0, world=4
2022-05-18T03:34:31.8737214Z dist init r=1, world=4
2022-05-18T03:34:31.9123218Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:34:31.9225077Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:34:31.9225619Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:34:31.9226531Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:34:31.9227216Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:31.9227803Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:31.9228392Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:31.9228922Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:31.9333352Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:34:31.9334047Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:34:31.9334491Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:34:31.9334839Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:34:32.1319212Z skip: Need at least 2 CUDA devices (0.915s)
2022-05-18T03:34:32.1337790Z   test_communication_nested_model_True_use_no_sync_False_sharding_strategy_None (__main__.TestCommunication)
2022-05-18T03:34:32.1373407Z Tests FSDP's communication cost in terms of calls to collective ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2770
2022-05-18T03:34:32.1398808Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2771
2022-05-18T03:34:32.1422405Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2772
2022-05-18T03:34:32.1445944Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2773
2022-05-18T03:34:32.7391793Z dist init r=1, world=4
2022-05-18T03:34:32.7510933Z dist init r=3, world=4
2022-05-18T03:34:32.7582671Z dist init r=2, world=4
2022-05-18T03:34:32.7865894Z dist init r=0, world=4
2022-05-18T03:34:32.8095322Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:34:32.8195563Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:34:32.8196927Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:34:32.8197651Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:32.8198075Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:34:32.8198574Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:32.8199208Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:32.8201659Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:32.8207011Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:34:32.8207663Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:34:32.8208217Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:34:32.8208704Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:34:33.0474934Z skip: Need at least 2 CUDA devices (0.915s)
2022-05-18T03:34:33.0523696Z   test_communication_nested_model_True_use_no_sync_False_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestCommunication)
2022-05-18T03:34:33.0562329Z Tests FSDP's communication cost in terms of calls to collective ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2826
2022-05-18T03:34:33.0586943Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2827
2022-05-18T03:34:33.0610247Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2828
2022-05-18T03:34:33.0632828Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2829
2022-05-18T03:34:33.7713332Z dist init r=0, world=4
2022-05-18T03:34:33.7868907Z dist init r=1, world=4
2022-05-18T03:34:33.8208679Z dist init r=3, world=4
2022-05-18T03:34:33.8529997Z dist init r=2, world=4
2022-05-18T03:34:33.8718312Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:34:33.8827407Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:34:33.8929699Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:34:33.8930360Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:34:33.8930976Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:33.8931496Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:33.8932023Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:33.9022802Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:33.9036670Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:34:33.9037954Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:34:33.9038776Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:34:33.9039574Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:34:34.0660394Z skip: Need at least 2 CUDA devices (1.018s)
2022-05-18T03:34:34.0679024Z   test_communication_nested_model_True_use_no_sync_True_sharding_strategy_None (__main__.TestCommunication)
2022-05-18T03:34:34.0715093Z Tests FSDP's communication cost in terms of calls to collective ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2882
2022-05-18T03:34:34.0741132Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2883
2022-05-18T03:34:34.0764184Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2884
2022-05-18T03:34:34.0787820Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2885
2022-05-18T03:34:34.6935455Z dist init r=2, world=4
2022-05-18T03:34:34.7030147Z dist init r=3, world=4
2022-05-18T03:34:34.7124088Z dist init r=1, world=4
2022-05-18T03:34:34.7129576Z dist init r=0, world=4
2022-05-18T03:34:34.7434973Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:34:34.7535984Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:34:34.7536956Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:34:34.7537823Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:34:34.7538877Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:34.7539641Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:34.7540438Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:34.7541243Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:34.7543511Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:34:34.7544166Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:34:34.7545681Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:34:34.7546786Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:34:34.9814303Z skip: Need at least 2 CUDA devices (0.915s)
2022-05-18T03:34:34.9832779Z   test_communication_nested_model_True_use_no_sync_True_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestCommunication)
2022-05-18T03:34:34.9868719Z Tests FSDP's communication cost in terms of calls to collective ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2938
2022-05-18T03:34:34.9894131Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2939
2022-05-18T03:34:34.9917671Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2940
2022-05-18T03:34:34.9941141Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2941
2022-05-18T03:34:35.5987350Z dist init r=0, world=4
2022-05-18T03:34:35.6274556Z dist init r=2, world=4
2022-05-18T03:34:35.6325022Z dist init r=1, world=4
2022-05-18T03:34:35.6419714Z dist init r=3, world=4
2022-05-18T03:34:35.6584658Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:34:35.6684730Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:34:35.6786113Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:34:35.6786681Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:34:35.6787577Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:35.6788376Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:35.6789042Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:35.6789815Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:35.6894067Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:34:35.6894855Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:34:35.6895455Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:34:35.6896099Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:34:35.8967436Z skip: Need at least 2 CUDA devices (0.915s)
2022-05-18T03:34:35.8967865Z 
2022-05-18T03:34:35.8968759Z ----------------------------------------------------------------------
2022-05-18T03:34:35.8969238Z Ran 8 tests in 7.703s
2022-05-18T03:34:35.8969384Z 
2022-05-18T03:34:35.8969706Z OK (skipped=8)
2022-05-18T03:34:35.8969824Z 
2022-05-18T03:34:35.8969898Z Generating XML reports...
2022-05-18T03:34:35.9010910Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_comm/TEST-TestCommunication-20220518033428.xml
2022-05-18T03:34:36.0901180Z Running distributed/fsdp/test_fsdp_core ... [2022-05-18 03:34:36.089690]
2022-05-18T03:34:36.0901724Z Executing ['/opt/conda/bin/python', 'distributed/fsdp/test_fsdp_core.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-05-18 03:34:36.089793]
2022-05-18T03:34:36.6487674Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5bs0mzyw
2022-05-18T03:34:36.6488582Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5bs0mzyw/_remote_module_non_scriptable.py
2022-05-18T03:34:36.6679888Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_core
2022-05-18T03:34:36.6710561Z 
2022-05-18T03:34:36.6710914Z Running tests...
2022-05-18T03:34:36.6711340Z ----------------------------------------------------------------------
2022-05-18T03:34:36.9504063Z   test_backward_hooks_after_save (__main__.TestHooks) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3005
2022-05-18T03:34:36.9525738Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3006
2022-05-18T03:34:36.9548600Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3007
2022-05-18T03:34:36.9571744Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3008
2022-05-18T03:34:37.6317540Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc92bh6at
2022-05-18T03:34:37.6318380Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc92bh6at/_remote_module_non_scriptable.py
2022-05-18T03:34:37.6479030Z dist init r=0, world=4
2022-05-18T03:34:37.7016343Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpukka42ej
2022-05-18T03:34:37.7017372Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpukka42ej/_remote_module_non_scriptable.py
2022-05-18T03:34:37.7029661Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeerafjbe
2022-05-18T03:34:37.7031900Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeerafjbe/_remote_module_non_scriptable.py
2022-05-18T03:34:37.7101572Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe2mel2p5
2022-05-18T03:34:37.7103804Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe2mel2p5/_remote_module_non_scriptable.py
2022-05-18T03:34:37.7177193Z dist init r=3, world=4
2022-05-18T03:34:37.7188289Z dist init r=2, world=4
2022-05-18T03:34:37.7259900Z dist init r=1, world=4
2022-05-18T03:34:37.7488162Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:34:37.7589111Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:34:37.7690733Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:34:37.7691308Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:34:37.7692498Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:37.7693088Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:37.7694170Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:37.7694847Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:37.7698823Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:34:37.7699985Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:34:37.7700849Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:34:37.7701565Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:34:37.9604029Z skip: Need at least 2 CUDA devices (1.289s)
2022-05-18T03:34:37.9644430Z   test_output_backward_hooks_cuda_first_False (__main__.TestHooks) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3061
2022-05-18T03:34:37.9670818Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3062
2022-05-18T03:34:37.9694695Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3063
2022-05-18T03:34:37.9718648Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3064
2022-05-18T03:34:38.5497383Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwsgi_2lc
2022-05-18T03:34:38.5498216Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwsgi_2lc/_remote_module_non_scriptable.py
2022-05-18T03:34:38.5619515Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxsqsti8k
2022-05-18T03:34:38.5620552Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxsqsti8k/_remote_module_non_scriptable.py
2022-05-18T03:34:38.5654289Z dist init r=0, world=4
2022-05-18T03:34:38.5776574Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpes8qez5j
2022-05-18T03:34:38.5778043Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpes8qez5j/_remote_module_non_scriptable.py
2022-05-18T03:34:38.5778514Z dist init r=3, world=4
2022-05-18T03:34:38.5915203Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6o2is65_
2022-05-18T03:34:38.5916860Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6o2is65_/_remote_module_non_scriptable.py
2022-05-18T03:34:38.5932872Z dist init r=2, world=4
2022-05-18T03:34:38.6069718Z dist init r=1, world=4
2022-05-18T03:34:38.6189592Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:34:38.6290350Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:34:38.6391965Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:34:38.6392516Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:34:38.6393349Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:38.6394081Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:38.6394882Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:38.6395404Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:38.6498751Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:34:38.6499149Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:34:38.6499743Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:34:38.6501771Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:34:38.8745800Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:34:38.8790802Z   test_output_backward_hooks_cuda_first_True (__main__.TestHooks) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3117
2022-05-18T03:34:38.8817073Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3118
2022-05-18T03:34:38.8840459Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3119
2022-05-18T03:34:38.8864196Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3120
2022-05-18T03:34:39.4461988Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj_yswluq
2022-05-18T03:34:39.4462708Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj_yswluq/_remote_module_non_scriptable.py
2022-05-18T03:34:39.4471269Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphkiwjuhb
2022-05-18T03:34:39.4473175Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphkiwjuhb/_remote_module_non_scriptable.py
2022-05-18T03:34:39.4620312Z dist init r=2, world=4
2022-05-18T03:34:39.4631267Z dist init r=1, world=4
2022-05-18T03:34:39.4961663Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwls2ooal
2022-05-18T03:34:39.4962577Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwls2ooal/_remote_module_non_scriptable.py
2022-05-18T03:34:39.5024520Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpocgmq6ks
2022-05-18T03:34:39.5026273Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpocgmq6ks/_remote_module_non_scriptable.py
2022-05-18T03:34:39.5118900Z dist init r=0, world=4
2022-05-18T03:34:39.5182602Z dist init r=3, world=4
2022-05-18T03:34:39.5393039Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:34:39.5595530Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:34:39.5697264Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:34:39.5698457Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:39.5699089Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:34:39.5699714Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:39.5700402Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:39.5701028Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:39.5706125Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:34:39.5706917Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:34:39.5707477Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:34:39.5708019Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:34:39.7890723Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:34:39.7903201Z   test_register_functions_called_cuda_first_False_mixed_precision_False (__main__.TestHooks)
2022-05-18T03:34:39.7938811Z Tests that _register_{pre|post}_backward_hooks called during forward. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3173
2022-05-18T03:34:39.7964384Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3174
2022-05-18T03:34:39.7987232Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3175
2022-05-18T03:34:39.8011004Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3176
2022-05-18T03:34:40.4162680Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_10ixyzt
2022-05-18T03:34:40.4163621Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_10ixyzt/_remote_module_non_scriptable.py
2022-05-18T03:34:40.4233952Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv778snqe
2022-05-18T03:34:40.4235569Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv778snqe/_remote_module_non_scriptable.py
2022-05-18T03:34:40.4313223Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9d3i3zb4
2022-05-18T03:34:40.4314610Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9d3i3zb4/_remote_module_non_scriptable.py
2022-05-18T03:34:40.4324514Z dist init r=0, world=4
2022-05-18T03:34:40.4394611Z dist init r=1, world=4
2022-05-18T03:34:40.4423131Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc3glphd2
2022-05-18T03:34:40.4424786Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc3glphd2/_remote_module_non_scriptable.py
2022-05-18T03:34:40.4473192Z dist init r=2, world=4
2022-05-18T03:34:40.4579215Z dist init r=3, world=4
2022-05-18T03:34:40.4686853Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:34:40.4787833Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:34:40.4890712Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:34:40.4891404Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:34:40.4893197Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:40.4894025Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:40.4894796Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:40.4895389Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:40.4897050Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:34:40.4898207Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:34:40.4898868Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:34:40.4899220Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:34:40.7037906Z skip: Need at least 2 CUDA devices (0.915s)
2022-05-18T03:34:40.7047673Z   test_register_functions_called_cuda_first_False_mixed_precision_True (__main__.TestHooks)
2022-05-18T03:34:40.7084160Z Tests that _register_{pre|post}_backward_hooks called during forward. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3229
2022-05-18T03:34:40.7109598Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3230
2022-05-18T03:34:40.7132492Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3231
2022-05-18T03:34:40.7156324Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3232
2022-05-18T03:34:41.3037684Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsho61jns
2022-05-18T03:34:41.3041691Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsho61jns/_remote_module_non_scriptable.py
2022-05-18T03:34:41.3195552Z dist init r=1, world=4
2022-05-18T03:34:41.3324430Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa8wkkr8d
2022-05-18T03:34:41.3324958Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_px0p2lu
2022-05-18T03:34:41.3326464Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_px0p2lu/_remote_module_non_scriptable.py
2022-05-18T03:34:41.3326899Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa8wkkr8d/_remote_module_non_scriptable.py
2022-05-18T03:34:41.3481266Z dist init r=0, world=4
2022-05-18T03:34:41.3481496Z dist init r=3, world=4
2022-05-18T03:34:41.3625244Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr7fz_i9o
2022-05-18T03:34:41.3627242Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr7fz_i9o/_remote_module_non_scriptable.py
2022-05-18T03:34:41.3780291Z dist init r=2, world=4
2022-05-18T03:34:41.3891654Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:34:41.4094459Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:34:41.4196156Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:34:41.4196884Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:34:41.4197927Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:41.4198908Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:41.4199590Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:41.4200216Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:41.4204365Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:34:41.4204919Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:34:41.4205450Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:34:41.4208270Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:34:41.6183173Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:34:41.6193017Z   test_register_functions_called_cuda_first_True_mixed_precision_False (__main__.TestHooks)
2022-05-18T03:34:41.6228361Z Tests that _register_{pre|post}_backward_hooks called during forward. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3285
2022-05-18T03:34:41.6254161Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3286
2022-05-18T03:34:41.6277263Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3287
2022-05-18T03:34:41.6301121Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3288
2022-05-18T03:34:42.2116967Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_s0x4ga_
2022-05-18T03:34:42.2117760Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_s0x4ga_/_remote_module_non_scriptable.py
2022-05-18T03:34:42.2275728Z dist init r=2, world=4
2022-05-18T03:34:42.2355836Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdhgqfv9c
2022-05-18T03:34:42.2358052Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdhgqfv9c/_remote_module_non_scriptable.py
2022-05-18T03:34:42.2485339Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbnzruznq
2022-05-18T03:34:42.2487214Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbnzruznq/_remote_module_non_scriptable.py
2022-05-18T03:34:42.2498701Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2rg2v4t8
2022-05-18T03:34:42.2501044Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2rg2v4t8/_remote_module_non_scriptable.py
2022-05-18T03:34:42.2517025Z dist init r=0, world=4
2022-05-18T03:34:42.2646193Z dist init r=1, world=4
2022-05-18T03:34:42.2660840Z dist init r=3, world=4
2022-05-18T03:34:42.2785884Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:34:42.2957576Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:34:42.3060081Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:34:42.3060649Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:34:42.3061283Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:42.3061814Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:42.3062323Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:42.3090445Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:42.3168546Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:34:42.3169042Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:34:42.3169571Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:34:42.3170083Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:34:42.5328694Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:34:42.5337526Z   test_register_functions_called_cuda_first_True_mixed_precision_True (__main__.TestHooks)
2022-05-18T03:34:42.5373530Z Tests that _register_{pre|post}_backward_hooks called during forward. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3341
2022-05-18T03:34:42.5400022Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3342
2022-05-18T03:34:42.5423169Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3343
2022-05-18T03:34:42.5447591Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3344
2022-05-18T03:34:43.1159281Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp21inu93i
2022-05-18T03:34:43.1160538Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp21inu93i/_remote_module_non_scriptable.py
2022-05-18T03:34:43.1302542Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuyorlu9f
2022-05-18T03:34:43.1303552Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuyorlu9f/_remote_module_non_scriptable.py
2022-05-18T03:34:43.1316476Z dist init r=2, world=4
2022-05-18T03:34:43.1458938Z dist init r=3, world=4
2022-05-18T03:34:43.1602932Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl5p02ren
2022-05-18T03:34:43.1605033Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl5p02ren/_remote_module_non_scriptable.py
2022-05-18T03:34:43.1759970Z dist init r=1, world=4
2022-05-18T03:34:43.1789645Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf3chelru
2022-05-18T03:34:43.1791885Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf3chelru/_remote_module_non_scriptable.py
2022-05-18T03:34:43.1946441Z dist init r=0, world=4
2022-05-18T03:34:43.2069746Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:34:43.2271238Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:34:43.2271972Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:34:43.2272660Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:34:43.2273641Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:43.2274345Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:43.2274879Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:43.2275624Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:43.2379839Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:34:43.2380227Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:34:43.2380727Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:34:43.2381264Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:34:43.4473803Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:34:43.4518760Z   test_transformer_no_grad_mixed_precision_False (__main__.TestNoGrad) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3397
2022-05-18T03:34:43.4544324Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3398
2022-05-18T03:34:43.4567612Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3399
2022-05-18T03:34:43.4591122Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3400
2022-05-18T03:34:44.0457334Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl2z189fx
2022-05-18T03:34:44.0458147Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl2z189fx/_remote_module_non_scriptable.py
2022-05-18T03:34:44.0551953Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp22lqkt6
2022-05-18T03:34:44.0553387Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp22lqkt6/_remote_module_non_scriptable.py
2022-05-18T03:34:44.0615277Z dist init r=1, world=4
2022-05-18T03:34:44.0640826Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4mkig3oy
2022-05-18T03:34:44.0642941Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4mkig3oy/_remote_module_non_scriptable.py
2022-05-18T03:34:44.0711621Z dist init r=3, world=4
2022-05-18T03:34:44.0798994Z dist init r=0, world=4
2022-05-18T03:34:44.0807106Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7pazasnd
2022-05-18T03:34:44.0809290Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7pazasnd/_remote_module_non_scriptable.py
2022-05-18T03:34:44.0962909Z dist init r=2, world=4
2022-05-18T03:34:44.1121367Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:34:44.1222107Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:34:44.1324186Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:34:44.1325180Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:44.1325847Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:34:44.1326554Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:44.1327320Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:44.1328123Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:44.1333827Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:34:44.1335377Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:34:44.1336254Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:34:44.1337017Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:34:44.3618023Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:34:44.3662092Z   test_transformer_no_grad_mixed_precision_True (__main__.TestNoGrad) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3453
2022-05-18T03:34:44.3687623Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3454
2022-05-18T03:34:44.3710676Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3455
2022-05-18T03:34:44.3734141Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3456
2022-05-18T03:34:44.9793880Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfjw55fcd
2022-05-18T03:34:44.9794666Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfjw55fcd/_remote_module_non_scriptable.py
2022-05-18T03:34:44.9870649Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuobi35wt
2022-05-18T03:34:44.9872083Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuobi35wt/_remote_module_non_scriptable.py
2022-05-18T03:34:44.9953122Z dist init r=0, world=4
2022-05-18T03:34:44.9977291Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbb089p4c
2022-05-18T03:34:44.9979090Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbb089p4c/_remote_module_non_scriptable.py
2022-05-18T03:34:45.0029033Z dist init r=2, world=4
2022-05-18T03:34:45.0068181Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2ivp8xki
2022-05-18T03:34:45.0070122Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2ivp8xki/_remote_module_non_scriptable.py
2022-05-18T03:34:45.0138439Z dist init r=3, world=4
2022-05-18T03:34:45.0225539Z dist init r=1, world=4
2022-05-18T03:34:45.0347188Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:34:45.0536496Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:34:45.0637682Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:34:45.0638310Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:34:45.0638994Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:45.0639513Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:45.0640038Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:45.0651242Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:45.0745679Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:34:45.0746358Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:34:45.0746819Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:34:45.0747360Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:34:45.2761541Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:34:45.2806468Z   test_param_change_after_init_mixed_precision_False (__main__.TestParamInit) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3509
2022-05-18T03:34:45.2832530Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3510
2022-05-18T03:34:45.2855482Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3511
2022-05-18T03:34:45.2879429Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3512
2022-05-18T03:34:45.8490843Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp07fd9i3k
2022-05-18T03:34:45.8491774Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp07fd9i3k/_remote_module_non_scriptable.py
2022-05-18T03:34:45.8496200Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppqtajb2t
2022-05-18T03:34:45.8498915Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppqtajb2t/_remote_module_non_scriptable.py
2022-05-18T03:34:45.8646789Z dist init r=2, world=4
2022-05-18T03:34:45.8656619Z dist init r=0, world=4
2022-05-18T03:34:45.8836605Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_a8_ukqr
2022-05-18T03:34:45.8837592Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_a8_ukqr/_remote_module_non_scriptable.py
2022-05-18T03:34:45.8993825Z dist init r=3, world=4
2022-05-18T03:34:45.9337213Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9ly5swh3
2022-05-18T03:34:45.9338448Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9ly5swh3/_remote_module_non_scriptable.py
2022-05-18T03:34:45.9494076Z dist init r=1, world=4
2022-05-18T03:34:45.9761165Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:34:45.9861553Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:34:45.9965118Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:34:45.9965848Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:45.9966394Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:34:45.9967085Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:45.9967615Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:45.9968156Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:46.0071431Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:34:46.0071867Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:34:46.0072404Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:34:46.0072775Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:34:46.1905752Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:34:46.1950354Z   test_param_change_after_init_mixed_precision_True (__main__.TestParamInit) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3565
2022-05-18T03:34:46.1976102Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3566
2022-05-18T03:34:46.1999512Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3567
2022-05-18T03:34:46.2023232Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3568
2022-05-18T03:34:46.8124944Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz83nqjpo
2022-05-18T03:34:46.8127460Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz83nqjpo/_remote_module_non_scriptable.py
2022-05-18T03:34:46.8177654Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp08_0bj1l
2022-05-18T03:34:46.8179405Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp08_0bj1l/_remote_module_non_scriptable.py
2022-05-18T03:34:46.8220513Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7ufvb0tj
2022-05-18T03:34:46.8223125Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7ufvb0tj/_remote_module_non_scriptable.py
2022-05-18T03:34:46.8265693Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpax2l3est
2022-05-18T03:34:46.8266685Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpax2l3est/_remote_module_non_scriptable.py
2022-05-18T03:34:46.8286675Z dist init r=2, world=4
2022-05-18T03:34:46.8340432Z dist init r=1, world=4
2022-05-18T03:34:46.8379602Z dist init r=0, world=4
2022-05-18T03:34:46.8421216Z dist init r=3, world=4
2022-05-18T03:34:46.8597003Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:34:46.8698266Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:34:46.8799874Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:34:46.8800630Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:46.8801199Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:34:46.8802004Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:46.8804437Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:46.8805318Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:46.8807551Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:34:46.8808585Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:34:46.8809236Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:34:46.8809884Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:34:47.1050382Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:34:47.1092999Z   test_delayed_optim_step_offload_false_none_no_shard (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3621
2022-05-18T03:34:47.1119048Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3622
2022-05-18T03:34:47.1142131Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3623
2022-05-18T03:34:47.1165651Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3624
2022-05-18T03:34:47.6716465Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcrdvexme
2022-05-18T03:34:47.6717579Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcrdvexme/_remote_module_non_scriptable.py
2022-05-18T03:34:47.6875087Z dist init r=1, world=4
2022-05-18T03:34:47.7195615Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl6o91ovl
2022-05-18T03:34:47.7196723Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl6o91ovl/_remote_module_non_scriptable.py
2022-05-18T03:34:47.7352315Z dist init r=0, world=4
2022-05-18T03:34:47.7656776Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp21c4yaud
2022-05-18T03:34:47.7657684Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp21c4yaud/_remote_module_non_scriptable.py
2022-05-18T03:34:47.7688191Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmissg_91
2022-05-18T03:34:47.7689832Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmissg_91/_remote_module_non_scriptable.py
2022-05-18T03:34:47.7810923Z dist init r=3, world=4
2022-05-18T03:34:47.7844268Z dist init r=2, world=4
2022-05-18T03:34:47.8054171Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:34:47.8155191Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:34:47.8256755Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:34:47.8257462Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:34:47.8258383Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:47.8259160Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:47.8259820Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:47.8260500Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:47.8265667Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:34:47.8266191Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:34:47.8266888Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:34:47.8267505Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:34:48.0192428Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:34:48.0235051Z   test_delayed_optim_step_offload_false_none_none (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3677
2022-05-18T03:34:48.0260623Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3678
2022-05-18T03:34:48.0284189Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3679
2022-05-18T03:34:48.0308148Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3680
2022-05-18T03:34:48.5894650Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvypun_15
2022-05-18T03:34:48.5895471Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvypun_15/_remote_module_non_scriptable.py
2022-05-18T03:34:48.6050738Z dist init r=1, world=4
2022-05-18T03:34:48.6205687Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1h3g2lmm
2022-05-18T03:34:48.6206825Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1h3g2lmm/_remote_module_non_scriptable.py
2022-05-18T03:34:48.6365156Z dist init r=2, world=4
2022-05-18T03:34:48.7233903Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf9mi1ff5
2022-05-18T03:34:48.7234590Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpci3gm37t
2022-05-18T03:34:48.7235168Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf9mi1ff5/_remote_module_non_scriptable.py
2022-05-18T03:34:48.7237019Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpci3gm37t/_remote_module_non_scriptable.py
2022-05-18T03:34:48.7387844Z dist init r=0, world=4
2022-05-18T03:34:48.7389853Z dist init r=3, world=4
2022-05-18T03:34:48.7600655Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:34:48.7668206Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:34:48.7700650Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:34:48.7701427Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:48.7701852Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:34:48.7702750Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:48.7703777Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:48.7770855Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:48.7809379Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:34:48.7810060Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:34:48.7810681Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:34:48.7811341Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:34:48.9334239Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:34:48.9375227Z   test_delayed_optim_step_offload_false_none_shard_grad_op (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3733
2022-05-18T03:34:48.9401285Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3734
2022-05-18T03:34:48.9424533Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3735
2022-05-18T03:34:48.9447965Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3736
2022-05-18T03:34:49.5289101Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp30758r78
2022-05-18T03:34:49.5289951Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9wee03xp
2022-05-18T03:34:49.5290615Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp30758r78/_remote_module_non_scriptable.py
2022-05-18T03:34:49.5291298Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9wee03xp/_remote_module_non_scriptable.py
2022-05-18T03:34:49.5446101Z dist init r=0, world=4
2022-05-18T03:34:49.5446441Z dist init r=3, world=4
2022-05-18T03:34:49.5548656Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2po4ac1p
2022-05-18T03:34:49.5550841Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2po4ac1p/_remote_module_non_scriptable.py
2022-05-18T03:34:49.5578639Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpglnjz4g_
2022-05-18T03:34:49.5580955Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpglnjz4g_/_remote_module_non_scriptable.py
2022-05-18T03:34:49.5704286Z dist init r=1, world=4
2022-05-18T03:34:49.5735880Z dist init r=2, world=4
2022-05-18T03:34:49.6015164Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:34:49.6015567Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:34:49.6116871Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:34:49.6119235Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:34:49.6120166Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:49.6120777Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:49.6121286Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:49.6121799Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:49.6223742Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:34:49.6224517Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:34:49.6225330Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:34:49.6225908Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:34:49.8475527Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:34:49.8517700Z   test_delayed_optim_step_offload_false_prefetch_post_no_shard (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3789
2022-05-18T03:34:49.8542781Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3790
2022-05-18T03:34:49.8566123Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3791
2022-05-18T03:34:49.8589768Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3792
2022-05-18T03:34:50.4680217Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnkvq_mv6
2022-05-18T03:34:50.4680997Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnkvq_mv6/_remote_module_non_scriptable.py
2022-05-18T03:34:50.4784727Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp__100doo
2022-05-18T03:34:50.4785502Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp__100doo/_remote_module_non_scriptable.py
2022-05-18T03:34:50.4839037Z dist init r=2, world=4
2022-05-18T03:34:50.4942021Z dist init r=3, world=4
2022-05-18T03:34:50.5052847Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuu2csg0u
2022-05-18T03:34:50.5053567Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuu2csg0u/_remote_module_non_scriptable.py
2022-05-18T03:34:50.5125070Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbr2d837g
2022-05-18T03:34:50.5126843Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbr2d837g/_remote_module_non_scriptable.py
2022-05-18T03:34:50.5214357Z dist init r=0, world=4
2022-05-18T03:34:50.5285698Z dist init r=1, world=4
2022-05-18T03:34:50.5453007Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:34:50.5553827Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:34:50.5654988Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:34:50.5655584Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:34:50.5656526Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:50.5657462Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:50.5658276Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:50.5658876Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:50.5762397Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:34:50.5763094Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:34:50.5763656Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:34:50.5764196Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:34:50.7616128Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:34:50.7659621Z   test_delayed_optim_step_offload_false_prefetch_post_none (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3845
2022-05-18T03:34:50.7685321Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3846
2022-05-18T03:34:50.7709046Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3847
2022-05-18T03:34:50.7732734Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3848
2022-05-18T03:34:51.3335138Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4kb83pe9
2022-05-18T03:34:51.3337790Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4kb83pe9/_remote_module_non_scriptable.py
2022-05-18T03:34:51.3495088Z dist init r=3, world=4
2022-05-18T03:34:51.3927981Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3m60hm9q
2022-05-18T03:34:51.3928732Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3m60hm9q/_remote_module_non_scriptable.py
2022-05-18T03:34:51.3945442Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprpw2p2s_
2022-05-18T03:34:51.3947688Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprpw2p2s_/_remote_module_non_scriptable.py
2022-05-18T03:34:51.4001109Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa2tnve10
2022-05-18T03:34:51.4003222Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa2tnve10/_remote_module_non_scriptable.py
2022-05-18T03:34:51.4089558Z dist init r=2, world=4
2022-05-18T03:34:51.4105340Z dist init r=0, world=4
2022-05-18T03:34:51.4158863Z dist init r=1, world=4
2022-05-18T03:34:51.4400453Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:34:51.4500732Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:34:51.4603520Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:51.4604115Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:34:51.4604911Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:51.4605486Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:34:51.4606077Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:51.4606621Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:51.4709842Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:34:51.4710515Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:34:51.4710991Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:34:51.4711652Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:34:51.6759349Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:34:51.6801713Z   test_delayed_optim_step_offload_false_prefetch_post_shard_grad_op (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3901
2022-05-18T03:34:51.6827350Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3902
2022-05-18T03:34:51.6850343Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3903
2022-05-18T03:34:51.6873905Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3904
2022-05-18T03:34:52.2545678Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplbovr74d
2022-05-18T03:34:52.2557949Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplbovr74d/_remote_module_non_scriptable.py
2022-05-18T03:34:52.2773278Z dist init r=2, world=4
2022-05-18T03:34:52.2937923Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxvbcskoz
2022-05-18T03:34:52.2940464Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxvbcskoz/_remote_module_non_scriptable.py
2022-05-18T03:34:52.3087514Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1rrcimz1
2022-05-18T03:34:52.3089433Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1rrcimz1/_remote_module_non_scriptable.py
2022-05-18T03:34:52.3100946Z dist init r=1, world=4
2022-05-18T03:34:52.3155515Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw5i5199h
2022-05-18T03:34:52.3170273Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw5i5199h/_remote_module_non_scriptable.py
2022-05-18T03:34:52.3257274Z dist init r=0, world=4
2022-05-18T03:34:52.3318869Z dist init r=3, world=4
2022-05-18T03:34:52.3529266Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:34:52.3529849Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:34:52.3630619Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:34:52.3631210Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:34:52.3631958Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:52.3632547Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:52.3633115Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:52.3633746Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:52.3737725Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:34:52.3738178Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:34:52.3738708Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:34:52.3739273Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:34:52.5900541Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:34:52.5946826Z   test_delayed_optim_step_offload_false_prefetch_pre_no_shard (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3957
2022-05-18T03:34:52.5973516Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3958
2022-05-18T03:34:52.6003888Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3959
2022-05-18T03:34:52.6032436Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3960
2022-05-18T03:34:53.1991760Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps9_w4q9y
2022-05-18T03:34:53.1992911Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps9_w4q9y/_remote_module_non_scriptable.py
2022-05-18T03:34:53.2153508Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp79ne7vn8
2022-05-18T03:34:53.2154238Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp79ne7vn8/_remote_module_non_scriptable.py
2022-05-18T03:34:53.2155097Z dist init r=0, world=4
2022-05-18T03:34:53.2311604Z dist init r=3, world=4
2022-05-18T03:34:53.2480647Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp67ps3d8_
2022-05-18T03:34:53.2482125Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp67ps3d8_/_remote_module_non_scriptable.py
2022-05-18T03:34:53.2549600Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp84rddx5c
2022-05-18T03:34:53.2551750Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp84rddx5c/_remote_module_non_scriptable.py
2022-05-18T03:34:53.2643237Z dist init r=1, world=4
2022-05-18T03:34:53.2706181Z dist init r=2, world=4
2022-05-18T03:34:53.2821573Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:34:53.2922036Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:34:53.3024815Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:34:53.3025440Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:34:53.3026196Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:53.3027851Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:53.3028533Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:53.3029043Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:53.3131708Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:34:53.3132318Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:34:53.3132835Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:34:53.3133169Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:34:53.5059255Z skip: Need at least 2 CUDA devices (0.916s)
2022-05-18T03:34:53.5100188Z   test_delayed_optim_step_offload_false_prefetch_pre_none (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4013
2022-05-18T03:34:53.5125614Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4014
2022-05-18T03:34:53.5149380Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4015
2022-05-18T03:34:53.5173210Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4016
2022-05-18T03:34:54.1106860Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmq9pap2l
2022-05-18T03:34:54.1109183Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmq9pap2l/_remote_module_non_scriptable.py
2022-05-18T03:34:54.1126005Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr6pwglrh
2022-05-18T03:34:54.1128594Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr6pwglrh/_remote_module_non_scriptable.py
2022-05-18T03:34:54.1265625Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa4qioc_w
2022-05-18T03:34:54.1268153Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa4qioc_w/_remote_module_non_scriptable.py
2022-05-18T03:34:54.1270904Z dist init r=3, world=4
2022-05-18T03:34:54.1290602Z dist init r=1, world=4
2022-05-18T03:34:54.1429983Z dist init r=2, world=4
2022-05-18T03:34:54.1667741Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp604_3lsy
2022-05-18T03:34:54.1668655Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp604_3lsy/_remote_module_non_scriptable.py
2022-05-18T03:34:54.1825755Z dist init r=0, world=4
2022-05-18T03:34:54.2238784Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:34:54.2239504Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:34:54.2240575Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:54.2241254Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:34:54.2241918Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:34:54.2242648Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:54.2246281Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:54.2247157Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:54.2247952Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:34:54.2248618Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:34:54.2249140Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:34:54.2249653Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:34:54.4200659Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:34:54.4243006Z   test_delayed_optim_step_offload_false_prefetch_pre_shard_grad_op (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4069
2022-05-18T03:34:54.4269767Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4070
2022-05-18T03:34:54.4293042Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4071
2022-05-18T03:34:54.4317394Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4072
2022-05-18T03:34:55.0067541Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp93fnyirh
2022-05-18T03:34:55.0068730Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp93fnyirh/_remote_module_non_scriptable.py
2022-05-18T03:34:55.0224941Z dist init r=3, world=4
2022-05-18T03:34:55.0229680Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp96de34hs
2022-05-18T03:34:55.0231597Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp96de34hs/_remote_module_non_scriptable.py
2022-05-18T03:34:55.0393094Z dist init r=1, world=4
2022-05-18T03:34:55.0591334Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgaw5kqui
2022-05-18T03:34:55.0593354Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgaw5kqui/_remote_module_non_scriptable.py
2022-05-18T03:34:55.0698372Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu_nb0gxg
2022-05-18T03:34:55.0700356Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu_nb0gxg/_remote_module_non_scriptable.py
2022-05-18T03:34:55.0749984Z dist init r=2, world=4
2022-05-18T03:34:55.0854485Z dist init r=0, world=4
2022-05-18T03:34:55.1265841Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:34:55.1367200Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:34:55.1367744Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:34:55.1368606Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:55.1369959Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:34:55.1370854Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:55.1371426Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:55.1371957Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:55.1378274Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:34:55.1378959Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:34:55.1379460Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:34:55.1379986Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:34:55.3343176Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:34:55.3386387Z   test_delayed_optim_step_offload_true_none_no_shard (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4125
2022-05-18T03:34:55.3412636Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4126
2022-05-18T03:34:55.3436922Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4127
2022-05-18T03:34:55.3460811Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4128
2022-05-18T03:34:55.9002362Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdicrwl8e
2022-05-18T03:34:55.9003058Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdicrwl8e/_remote_module_non_scriptable.py
2022-05-18T03:34:55.9045392Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7pt5o94b
2022-05-18T03:34:55.9046999Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7pt5o94b/_remote_module_non_scriptable.py
2022-05-18T03:34:55.9162077Z dist init r=2, world=4
2022-05-18T03:34:55.9203767Z dist init r=0, world=4
2022-05-18T03:34:55.9484767Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdc_4nsxs
2022-05-18T03:34:55.9486442Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdc_4nsxs/_remote_module_non_scriptable.py
2022-05-18T03:34:55.9546590Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt4w_p80e
2022-05-18T03:34:55.9548431Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt4w_p80e/_remote_module_non_scriptable.py
2022-05-18T03:34:55.9645267Z dist init r=1, world=4
2022-05-18T03:34:55.9700707Z dist init r=3, world=4
2022-05-18T03:34:55.9956431Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:34:56.0057685Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:34:56.0058268Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:34:56.0058815Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:34:56.0059792Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:56.0060486Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:56.0061147Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:56.0061871Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:56.0066730Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:34:56.0067260Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:34:56.0068088Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:34:56.0068812Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:34:56.2487168Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:34:56.2528857Z   test_delayed_optim_step_offload_true_none_none (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4181
2022-05-18T03:34:56.2554726Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4182
2022-05-18T03:34:56.2578536Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4183
2022-05-18T03:34:56.2602325Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4184
2022-05-18T03:34:56.8546422Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxlhea2qy
2022-05-18T03:34:56.8548118Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxlhea2qy/_remote_module_non_scriptable.py
2022-05-18T03:34:56.8635876Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbcv44mhh
2022-05-18T03:34:56.8637238Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbcv44mhh/_remote_module_non_scriptable.py
2022-05-18T03:34:56.8706522Z dist init r=3, world=4
2022-05-18T03:34:56.8714135Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpshioy3rs
2022-05-18T03:34:56.8716095Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpshioy3rs/_remote_module_non_scriptable.py
2022-05-18T03:34:56.8794616Z dist init r=2, world=4
2022-05-18T03:34:56.8849123Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1ao2ijhj
2022-05-18T03:34:56.8851476Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1ao2ijhj/_remote_module_non_scriptable.py
2022-05-18T03:34:56.8872798Z dist init r=1, world=4
2022-05-18T03:34:56.9006521Z dist init r=0, world=4
2022-05-18T03:34:56.9419199Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:34:56.9520849Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:34:56.9521790Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:56.9522529Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:34:56.9523108Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:34:56.9523877Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:56.9524664Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:56.9525443Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:56.9530857Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:34:56.9531545Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:34:56.9531961Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:34:56.9532298Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:34:57.1629224Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:34:57.1671996Z   test_delayed_optim_step_offload_true_none_shard_grad_op (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4237
2022-05-18T03:34:57.1698244Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4238
2022-05-18T03:34:57.1721741Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4239
2022-05-18T03:34:57.1745912Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4240
2022-05-18T03:34:57.7703884Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_uhg_jzr
2022-05-18T03:34:57.7705149Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_uhg_jzr/_remote_module_non_scriptable.py
2022-05-18T03:34:57.7740347Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc9or2xw5
2022-05-18T03:34:57.7742123Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc9or2xw5/_remote_module_non_scriptable.py
2022-05-18T03:34:57.7861904Z dist init r=1, world=4
2022-05-18T03:34:57.7900230Z dist init r=2, world=4
2022-05-18T03:34:57.8315255Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyfd4rnw7
2022-05-18T03:34:57.8316052Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1pv68ny4
2022-05-18T03:34:57.8316815Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyfd4rnw7/_remote_module_non_scriptable.py
2022-05-18T03:34:57.8317501Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1pv68ny4/_remote_module_non_scriptable.py
2022-05-18T03:34:57.8470667Z dist init r=0, world=4
2022-05-18T03:34:57.8471059Z dist init r=3, world=4
2022-05-18T03:34:57.8674198Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:34:57.8682052Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:34:57.8782426Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:34:57.8783011Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:34:57.8783817Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:57.8784521Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:57.8785051Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:57.8877213Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:57.8891266Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:34:57.8891813Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:34:57.8892364Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:34:57.8892871Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:34:58.0772386Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:34:58.0815279Z   test_delayed_optim_step_offload_true_prefetch_post_no_shard (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4293
2022-05-18T03:34:58.0841683Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4294
2022-05-18T03:34:58.0865729Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4295
2022-05-18T03:34:58.0889960Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4296
2022-05-18T03:34:58.6863384Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpahkkdsop
2022-05-18T03:34:58.6864597Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpahkkdsop/_remote_module_non_scriptable.py
2022-05-18T03:34:58.6945231Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4pooju4y
2022-05-18T03:34:58.6947447Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4pooju4y/_remote_module_non_scriptable.py
2022-05-18T03:34:58.7021435Z dist init r=1, world=4
2022-05-18T03:34:58.7080145Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnlxibn1u
2022-05-18T03:34:58.7082061Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnlxibn1u/_remote_module_non_scriptable.py
2022-05-18T03:34:58.7086241Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppsclhdlz
2022-05-18T03:34:58.7088284Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppsclhdlz/_remote_module_non_scriptable.py
2022-05-18T03:34:58.7104011Z dist init r=3, world=4
2022-05-18T03:34:58.7238222Z dist init r=2, world=4
2022-05-18T03:34:58.7245233Z dist init r=0, world=4
2022-05-18T03:34:58.7549052Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:34:58.7649788Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:34:58.7752783Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:34:58.7753614Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:58.7754182Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:34:58.7754702Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:58.7755369Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:58.7756148Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:58.7760103Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:34:58.7761136Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:34:58.7761912Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:34:58.7762505Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:34:58.9916674Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:34:58.9959431Z   test_delayed_optim_step_offload_true_prefetch_post_none (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4349
2022-05-18T03:34:58.9984987Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4350
2022-05-18T03:34:59.0008383Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4351
2022-05-18T03:34:59.0032564Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4352
2022-05-18T03:34:59.5885807Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnlgeepbv
2022-05-18T03:34:59.5886566Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqyukoc09
2022-05-18T03:34:59.5887276Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnlgeepbv/_remote_module_non_scriptable.py
2022-05-18T03:34:59.5888001Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqyukoc09/_remote_module_non_scriptable.py
2022-05-18T03:34:59.6040496Z dist init r=2, world=4
2022-05-18T03:34:59.6040817Z dist init r=1, world=4
2022-05-18T03:34:59.6119500Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc3pdd7gr
2022-05-18T03:34:59.6121052Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc3pdd7gr/_remote_module_non_scriptable.py
2022-05-18T03:34:59.6276613Z dist init r=3, world=4
2022-05-18T03:34:59.6300546Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppf8m810r
2022-05-18T03:34:59.6302747Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppf8m810r/_remote_module_non_scriptable.py
2022-05-18T03:34:59.6455502Z dist init r=0, world=4
2022-05-18T03:34:59.6752509Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:34:59.6853266Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:34:59.6955149Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:34:59.6956330Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:59.6957427Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:34:59.6958229Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:59.6958961Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:59.6959555Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:34:59.6963441Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:34:59.6964128Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:34:59.6964611Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:34:59.6965093Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:34:59.9058538Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:34:59.9100473Z   test_delayed_optim_step_offload_true_prefetch_post_shard_grad_op (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4405
2022-05-18T03:34:59.9126491Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4406
2022-05-18T03:34:59.9149972Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4407
2022-05-18T03:34:59.9173277Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4408
2022-05-18T03:35:00.5170686Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4fb7rfok
2022-05-18T03:35:00.5171433Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4fb7rfok/_remote_module_non_scriptable.py
2022-05-18T03:35:00.5250177Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpayr3yhwd
2022-05-18T03:35:00.5250923Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpayr3yhwd/_remote_module_non_scriptable.py
2022-05-18T03:35:00.5333642Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphma8o3_m
2022-05-18T03:35:00.5334403Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphma8o3_m/_remote_module_non_scriptable.py
2022-05-18T03:35:00.5341700Z dist init r=1, world=4
2022-05-18T03:35:00.5420118Z dist init r=0, world=4
2022-05-18T03:35:00.5500628Z dist init r=2, world=4
2022-05-18T03:35:00.5676660Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9c52dffy
2022-05-18T03:35:00.5678027Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9c52dffy/_remote_module_non_scriptable.py
2022-05-18T03:35:00.5832810Z dist init r=3, world=4
2022-05-18T03:35:00.6013241Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:35:00.6013639Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:35:00.6114641Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:35:00.6115814Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:35:00.6116781Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:00.6117483Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:00.6118013Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:00.6118537Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:00.6222631Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:35:00.6223447Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:35:00.6223987Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:35:00.6224767Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:35:00.8200349Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:35:00.8241536Z   test_delayed_optim_step_offload_true_prefetch_pre_no_shard (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4461
2022-05-18T03:35:00.8267488Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4462
2022-05-18T03:35:00.8290678Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4463
2022-05-18T03:35:00.8314934Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4464
2022-05-18T03:35:01.4018327Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp46sxrgoy
2022-05-18T03:35:01.4019116Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp46sxrgoy/_remote_module_non_scriptable.py
2022-05-18T03:35:01.4177903Z dist init r=3, world=4
2022-05-18T03:35:01.4499453Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzadmxroj
2022-05-18T03:35:01.4500281Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzadmxroj/_remote_module_non_scriptable.py
2022-05-18T03:35:01.4655691Z dist init r=2, world=4
2022-05-18T03:35:01.4971068Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8antdfj3
2022-05-18T03:35:01.4972086Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8antdfj3/_remote_module_non_scriptable.py
2022-05-18T03:35:01.5049941Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl3vo2npb
2022-05-18T03:35:01.5051180Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl3vo2npb/_remote_module_non_scriptable.py
2022-05-18T03:35:01.5132768Z dist init r=1, world=4
2022-05-18T03:35:01.5206947Z dist init r=0, world=4
2022-05-18T03:35:01.5619226Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:35:01.5720246Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:35:01.5822600Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:35:01.5823424Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:35:01.5824290Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:01.5824997Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:01.5825519Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:01.5826044Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:01.5832852Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:35:01.5835964Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:35:01.5836395Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:35:01.5836811Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:35:01.7341360Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:35:01.7383686Z   test_delayed_optim_step_offload_true_prefetch_pre_none (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4517
2022-05-18T03:35:01.7409998Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4518
2022-05-18T03:35:01.7433456Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4519
2022-05-18T03:35:01.7457584Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4520
2022-05-18T03:35:02.3147931Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv7sq4c3u
2022-05-18T03:35:02.3148972Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv7sq4c3u/_remote_module_non_scriptable.py
2022-05-18T03:35:02.3149756Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprqhxe5j0
2022-05-18T03:35:02.3152973Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprqhxe5j0/_remote_module_non_scriptable.py
2022-05-18T03:35:02.3248647Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxc0a13eo
2022-05-18T03:35:02.3251213Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxc0a13eo/_remote_module_non_scriptable.py
2022-05-18T03:35:02.3315653Z dist init r=3, world=4
2022-05-18T03:35:02.3316016Z dist init r=2, world=4
2022-05-18T03:35:02.3416811Z dist init r=0, world=4
2022-05-18T03:35:02.3814944Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx3227bcb
2022-05-18T03:35:02.3816056Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx3227bcb/_remote_module_non_scriptable.py
2022-05-18T03:35:02.3969958Z dist init r=1, world=4
2022-05-18T03:35:02.4332412Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:35:02.4432300Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:35:02.4534353Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:35:02.4535120Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:35:02.4536206Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:02.4537188Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:02.4538173Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:02.4538776Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:02.4543138Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:35:02.4543623Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:35:02.4544025Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:35:02.4544367Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:35:02.6484223Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:35:02.6526115Z   test_delayed_optim_step_offload_true_prefetch_pre_shard_grad_op (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4573
2022-05-18T03:35:02.6551819Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4574
2022-05-18T03:35:02.6575589Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4575
2022-05-18T03:35:02.6599403Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4576
2022-05-18T03:35:03.3053407Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpszr0io68
2022-05-18T03:35:03.3054756Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpszr0io68/_remote_module_non_scriptable.py
2022-05-18T03:35:03.3196487Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpiedsz_cm
2022-05-18T03:35:03.3197454Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpiedsz_cm/_remote_module_non_scriptable.py
2022-05-18T03:35:03.3215673Z dist init r=0, world=4
2022-05-18T03:35:03.3352926Z dist init r=3, world=4
2022-05-18T03:35:03.3497050Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo49pkfbs
2022-05-18T03:35:03.3499248Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo49pkfbs/_remote_module_non_scriptable.py
2022-05-18T03:35:03.3655284Z dist init r=2, world=4
2022-05-18T03:35:03.3919008Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp29w8zp3p
2022-05-18T03:35:03.3920271Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp29w8zp3p/_remote_module_non_scriptable.py
2022-05-18T03:35:03.4083102Z dist init r=1, world=4
2022-05-18T03:35:03.4435214Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:35:03.4436140Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:35:03.4437146Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:35:03.4437894Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:35:03.4438760Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:03.4439592Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:03.4440402Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:03.4441154Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:03.4441716Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:35:03.4442267Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:35:03.4443542Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:35:03.4448060Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:35:03.6627997Z skip: Need at least 2 CUDA devices (1.014s)
2022-05-18T03:35:03.6671749Z   test_delayed_reduce_scatter_offload_false_none_no_shard (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4629
2022-05-18T03:35:03.6698475Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4630
2022-05-18T03:35:03.6722363Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4631
2022-05-18T03:35:03.6746546Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4632
2022-05-18T03:35:04.2947688Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_rf89huu
2022-05-18T03:35:04.2948430Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_rf89huu/_remote_module_non_scriptable.py
2022-05-18T03:35:04.2949124Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpofxkbzww
2022-05-18T03:35:04.2952825Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpofxkbzww/_remote_module_non_scriptable.py
2022-05-18T03:35:04.3105335Z dist init r=3, world=4
2022-05-18T03:35:04.3115514Z dist init r=2, world=4
2022-05-18T03:35:04.3173027Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjgiwsq3w
2022-05-18T03:35:04.3174859Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjgiwsq3w/_remote_module_non_scriptable.py
2022-05-18T03:35:04.3330755Z dist init r=1, world=4
2022-05-18T03:35:04.3394336Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe9gjag9u
2022-05-18T03:35:04.3396534Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe9gjag9u/_remote_module_non_scriptable.py
2022-05-18T03:35:04.3551783Z dist init r=0, world=4
2022-05-18T03:35:04.3828931Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:35:04.3929346Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:35:04.4031689Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:35:04.4032294Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:35:04.4033238Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:04.4034067Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:04.4034646Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:04.4037958Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:04.4043557Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:35:04.4044126Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:35:04.4044626Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:35:04.4046112Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:35:04.5773245Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:35:04.5814813Z   test_delayed_reduce_scatter_offload_false_none_none (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4685
2022-05-18T03:35:04.5841146Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4686
2022-05-18T03:35:04.5865008Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4687
2022-05-18T03:35:04.5888869Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4688
2022-05-18T03:35:05.2150564Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0ktmi2x0
2022-05-18T03:35:05.2151367Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0ktmi2x0/_remote_module_non_scriptable.py
2022-05-18T03:35:05.2292392Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpoq6h21ms
2022-05-18T03:35:05.2294379Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg8bl1al9
2022-05-18T03:35:05.2295155Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpoq6h21ms/_remote_module_non_scriptable.py
2022-05-18T03:35:05.2295891Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg8bl1al9/_remote_module_non_scriptable.py
2022-05-18T03:35:05.2311949Z dist init r=1, world=4
2022-05-18T03:35:05.2318502Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjb8hv1r0
2022-05-18T03:35:05.2320327Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjb8hv1r0/_remote_module_non_scriptable.py
2022-05-18T03:35:05.2451127Z dist init r=3, world=4
2022-05-18T03:35:05.2452730Z dist init r=0, world=4
2022-05-18T03:35:05.2475973Z dist init r=2, world=4
2022-05-18T03:35:05.2660494Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:35:05.2785967Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:35:05.2887651Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:35:05.2888103Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:35:05.2888729Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:05.2889247Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:05.2889998Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:05.2964201Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:05.2994874Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:35:05.2995416Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:35:05.2995972Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:35:05.2996537Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:35:05.4916302Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:35:05.4959184Z   test_delayed_reduce_scatter_offload_false_none_shard_grad_op (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4741
2022-05-18T03:35:05.4985087Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4742
2022-05-18T03:35:05.5008125Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4743
2022-05-18T03:35:05.5032313Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4744
2022-05-18T03:35:06.0855988Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj_6aanmc
2022-05-18T03:35:06.0856753Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj_6aanmc/_remote_module_non_scriptable.py
2022-05-18T03:35:06.1015937Z dist init r=1, world=4
2022-05-18T03:35:06.1112770Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsh6gmo5q
2022-05-18T03:35:06.1114110Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsh6gmo5q/_remote_module_non_scriptable.py
2022-05-18T03:35:06.1211069Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphj8m7ijm
2022-05-18T03:35:06.1212599Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphj8m7ijm/_remote_module_non_scriptable.py
2022-05-18T03:35:06.1216974Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4q1ly3mh
2022-05-18T03:35:06.1219448Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4q1ly3mh/_remote_module_non_scriptable.py
2022-05-18T03:35:06.1272289Z dist init r=0, world=4
2022-05-18T03:35:06.1371686Z dist init r=3, world=4
2022-05-18T03:35:06.1374876Z dist init r=2, world=4
2022-05-18T03:35:06.1627850Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:35:06.1628243Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:35:06.1728919Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:35:06.1729686Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:35:06.1730747Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:06.1731643Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:06.1732163Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:06.1732830Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:06.1835823Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:35:06.1836502Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:35:06.1837378Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:35:06.1837967Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:35:06.4058761Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:35:06.4101256Z   test_delayed_reduce_scatter_offload_false_prefetch_post_no_shard (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4797
2022-05-18T03:35:06.4127968Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4798
2022-05-18T03:35:06.4151812Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4799
2022-05-18T03:35:06.4176186Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4800
2022-05-18T03:35:07.0171172Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9k3ew95q
2022-05-18T03:35:07.0172050Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9k3ew95q/_remote_module_non_scriptable.py
2022-05-18T03:35:07.0232448Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa037jagr
2022-05-18T03:35:07.0233414Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa037jagr/_remote_module_non_scriptable.py
2022-05-18T03:35:07.0328965Z dist init r=1, world=4
2022-05-18T03:35:07.0389272Z dist init r=2, world=4
2022-05-18T03:35:07.0431526Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo_8j3lqp
2022-05-18T03:35:07.0433462Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo_8j3lqp/_remote_module_non_scriptable.py
2022-05-18T03:35:07.0586649Z dist init r=0, world=4
2022-05-18T03:35:07.0682200Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqq3evhjj
2022-05-18T03:35:07.0684535Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqq3evhjj/_remote_module_non_scriptable.py
2022-05-18T03:35:07.0838161Z dist init r=3, world=4
2022-05-18T03:35:07.0945448Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:35:07.1042617Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:35:07.1043189Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:35:07.1043567Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:35:07.1044236Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:07.1044818Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:07.1045376Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:07.1047624Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:07.1149573Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:35:07.1150265Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:35:07.1150945Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:35:07.1151469Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:35:07.3203616Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:35:07.3246699Z   test_delayed_reduce_scatter_offload_false_prefetch_post_none (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4853
2022-05-18T03:35:07.3273975Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4854
2022-05-18T03:35:07.3298489Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4855
2022-05-18T03:35:07.3323039Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4856
2022-05-18T03:35:07.9320326Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv5s__9dk
2022-05-18T03:35:07.9321176Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv5s__9dk/_remote_module_non_scriptable.py
2022-05-18T03:35:07.9459305Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz2nfsm1q
2022-05-18T03:35:07.9460545Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz2nfsm1q/_remote_module_non_scriptable.py
2022-05-18T03:35:07.9482673Z dist init r=0, world=4
2022-05-18T03:35:07.9617280Z dist init r=3, world=4
2022-05-18T03:35:07.9696400Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf4_us2gk
2022-05-18T03:35:07.9698407Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj36qdq2e
2022-05-18T03:35:07.9699100Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf4_us2gk/_remote_module_non_scriptable.py
2022-05-18T03:35:07.9700023Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj36qdq2e/_remote_module_non_scriptable.py
2022-05-18T03:35:07.9856563Z dist init r=2, world=4
2022-05-18T03:35:07.9860590Z dist init r=1, world=4
2022-05-18T03:35:08.0196467Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:35:08.0298109Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:35:08.0298685Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:35:08.0299247Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:35:08.0299992Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:08.0300544Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:08.0301075Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:08.0301701Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:08.0405562Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:35:08.0406231Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:35:08.0406895Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:35:08.0407308Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:35:08.2349218Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:35:08.2391476Z   test_delayed_reduce_scatter_offload_false_prefetch_post_shard_grad_op (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4909
2022-05-18T03:35:08.2417097Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4910
2022-05-18T03:35:08.2441476Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4911
2022-05-18T03:35:08.2465145Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4912
2022-05-18T03:35:08.8388632Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfps_xk37
2022-05-18T03:35:08.8389712Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfps_xk37/_remote_module_non_scriptable.py
2022-05-18T03:35:08.8535993Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfzzrssuo
2022-05-18T03:35:08.8536778Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfzzrssuo/_remote_module_non_scriptable.py
2022-05-18T03:35:08.8547482Z dist init r=2, world=4
2022-05-18T03:35:08.8621001Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph9hj1_ft
2022-05-18T03:35:08.8622099Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph9hj1_ft/_remote_module_non_scriptable.py
2022-05-18T03:35:08.8695923Z dist init r=3, world=4
2022-05-18T03:35:08.8778259Z dist init r=1, world=4
2022-05-18T03:35:08.8795340Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyzscow4b
2022-05-18T03:35:08.8797318Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyzscow4b/_remote_module_non_scriptable.py
2022-05-18T03:35:08.8952000Z dist init r=0, world=4
2022-05-18T03:35:08.9106770Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:35:08.9208187Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:35:08.9310819Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:35:08.9311439Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:35:08.9312338Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:08.9312945Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:08.9313516Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:08.9314090Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:08.9317785Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:35:08.9318356Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:35:08.9318938Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:35:08.9320672Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:35:09.1492417Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:35:09.1534972Z   test_delayed_reduce_scatter_offload_false_prefetch_pre_no_shard (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4965
2022-05-18T03:35:09.1560836Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4966
2022-05-18T03:35:09.1583702Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4967
2022-05-18T03:35:09.1608058Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4968
2022-05-18T03:35:09.7252621Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplpycd64a
2022-05-18T03:35:09.7253388Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplpycd64a/_remote_module_non_scriptable.py
2022-05-18T03:35:09.7260700Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2lic8twe
2022-05-18T03:35:09.7262237Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2lic8twe/_remote_module_non_scriptable.py
2022-05-18T03:35:09.7408903Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnj4falcj
2022-05-18T03:35:09.7409755Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnj4falcj/_remote_module_non_scriptable.py
2022-05-18T03:35:09.7416345Z dist init r=0, world=4
2022-05-18T03:35:09.7421664Z dist init r=2, world=4
2022-05-18T03:35:09.7569826Z dist init r=3, world=4
2022-05-18T03:35:09.7885137Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppyr9cskf
2022-05-18T03:35:09.7887142Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppyr9cskf/_remote_module_non_scriptable.py
2022-05-18T03:35:09.8039104Z dist init r=1, world=4
2022-05-18T03:35:09.8335122Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:35:09.8435521Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:35:09.8537550Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:35:09.8538189Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:35:09.8539031Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:09.8539570Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:09.8540148Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:09.8540680Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:09.8643843Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:35:09.8644395Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:35:09.8644905Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:35:09.8645455Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:35:10.0635435Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:35:10.0678345Z   test_delayed_reduce_scatter_offload_false_prefetch_pre_none (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5021
2022-05-18T03:35:10.0705675Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5022
2022-05-18T03:35:10.0729280Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5023
2022-05-18T03:35:10.0753336Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5024
2022-05-18T03:35:10.6669208Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp92a5bkam
2022-05-18T03:35:10.6670175Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp92a5bkam/_remote_module_non_scriptable.py
2022-05-18T03:35:10.6746640Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpoawkx9rx
2022-05-18T03:35:10.6748192Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpoawkx9rx/_remote_module_non_scriptable.py
2022-05-18T03:35:10.6831045Z dist init r=2, world=4
2022-05-18T03:35:10.6905205Z dist init r=0, world=4
2022-05-18T03:35:10.7005969Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7trhh92t
2022-05-18T03:35:10.7007940Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7trhh92t/_remote_module_non_scriptable.py
2022-05-18T03:35:10.7074931Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeazcj5ij
2022-05-18T03:35:10.7076875Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeazcj5ij/_remote_module_non_scriptable.py
2022-05-18T03:35:10.7163558Z dist init r=1, world=4
2022-05-18T03:35:10.7231327Z dist init r=3, world=4
2022-05-18T03:35:10.7442314Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:35:10.7543423Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:35:10.7645122Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:35:10.7645795Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:35:10.7646830Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:10.7647950Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:10.7648756Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:10.7649527Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:10.7654196Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:35:10.7654736Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:35:10.7655262Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:35:10.7656063Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:35:10.9780580Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:35:10.9823672Z   test_delayed_reduce_scatter_offload_false_prefetch_pre_shard_grad_op (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5077
2022-05-18T03:35:10.9849430Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5078
2022-05-18T03:35:10.9873521Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5079
2022-05-18T03:35:10.9897393Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5080
2022-05-18T03:35:11.5807236Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpye2ma_9o
2022-05-18T03:35:11.5808466Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpye2ma_9o/_remote_module_non_scriptable.py
2022-05-18T03:35:11.5960738Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuy5tkmqr
2022-05-18T03:35:11.5961813Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuy5tkmqr/_remote_module_non_scriptable.py
2022-05-18T03:35:11.5965125Z dist init r=3, world=4
2022-05-18T03:35:11.6120339Z dist init r=2, world=4
2022-05-18T03:35:11.6156009Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpma2f96w6
2022-05-18T03:35:11.6157793Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpma2f96w6/_remote_module_non_scriptable.py
2022-05-18T03:35:11.6216013Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm9748_0k
2022-05-18T03:35:11.6217967Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm9748_0k/_remote_module_non_scriptable.py
2022-05-18T03:35:11.6312496Z dist init r=1, world=4
2022-05-18T03:35:11.6373908Z dist init r=0, world=4
2022-05-18T03:35:11.6576150Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:35:11.6732129Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:35:11.6834322Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:35:11.6835002Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:11.6835418Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:35:11.6835919Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:11.6836445Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:11.6880245Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:11.6941603Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:35:11.6942281Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:35:11.6943139Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:35:11.6943777Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:35:11.8923985Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:35:11.8966561Z   test_delayed_reduce_scatter_offload_true_none_no_shard (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5133
2022-05-18T03:35:11.8991968Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5134
2022-05-18T03:35:11.9015253Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5135
2022-05-18T03:35:11.9039494Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5136
2022-05-18T03:35:12.4898897Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6psv8jb5
2022-05-18T03:35:12.4899922Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6psv8jb5/_remote_module_non_scriptable.py
2022-05-18T03:35:12.4932898Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_qgiovol
2022-05-18T03:35:12.4934351Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_qgiovol/_remote_module_non_scriptable.py
2022-05-18T03:35:12.5030268Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4_pib0_d
2022-05-18T03:35:12.5031819Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4_pib0_d/_remote_module_non_scriptable.py
2022-05-18T03:35:12.5061901Z dist init r=2, world=4
2022-05-18T03:35:12.5093658Z dist init r=1, world=4
2022-05-18T03:35:12.5189543Z dist init r=0, world=4
2022-05-18T03:35:12.5191118Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbyjp_om5
2022-05-18T03:35:12.5193273Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbyjp_om5/_remote_module_non_scriptable.py
2022-05-18T03:35:12.5346871Z dist init r=3, world=4
2022-05-18T03:35:12.5472877Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:35:12.5576006Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:35:12.5576535Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:35:12.5577239Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:35:12.5578220Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:12.5578761Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:12.5579281Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:12.5579807Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:12.5588277Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:35:12.5588860Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:35:12.5589492Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:35:12.5590035Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:35:12.7064409Z skip: Need at least 2 CUDA devices (0.814s)
2022-05-18T03:35:12.7106598Z   test_delayed_reduce_scatter_offload_true_none_none (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5189
2022-05-18T03:35:12.7132252Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5190
2022-05-18T03:35:12.7155806Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5191
2022-05-18T03:35:12.7179239Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5192
2022-05-18T03:35:13.2851167Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq0eufgo1
2022-05-18T03:35:13.2852170Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq0eufgo1/_remote_module_non_scriptable.py
2022-05-18T03:35:13.2852876Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwys7cu1k
2022-05-18T03:35:13.2854191Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwys7cu1k/_remote_module_non_scriptable.py
2022-05-18T03:35:13.3010717Z dist init r=3, world=4
2022-05-18T03:35:13.3011492Z dist init r=1, world=4
2022-05-18T03:35:13.3209104Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkmi5rkio
2022-05-18T03:35:13.3210319Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkmi5rkio/_remote_module_non_scriptable.py
2022-05-18T03:35:13.3364855Z dist init r=2, world=4
2022-05-18T03:35:13.3490532Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc0i0423h
2022-05-18T03:35:13.3491951Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc0i0423h/_remote_module_non_scriptable.py
2022-05-18T03:35:13.3646954Z dist init r=0, world=4
2022-05-18T03:35:13.3925203Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:35:13.4026207Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:35:13.4128208Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:35:13.4129422Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:13.4130080Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:35:13.4130798Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:13.4131525Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:13.4132322Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:13.4136031Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:35:13.4136702Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:35:13.4137377Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:35:13.4137782Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:35:13.6206534Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:35:13.6248827Z   test_delayed_reduce_scatter_offload_true_none_shard_grad_op (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5245
2022-05-18T03:35:13.6274327Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5246
2022-05-18T03:35:13.6296979Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5247
2022-05-18T03:35:13.6320953Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5248
2022-05-18T03:35:14.2237169Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwz6kymji
2022-05-18T03:35:14.2237910Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwz6kymji/_remote_module_non_scriptable.py
2022-05-18T03:35:14.2375480Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv88nr3xx
2022-05-18T03:35:14.2377800Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv88nr3xx/_remote_module_non_scriptable.py
2022-05-18T03:35:14.2398747Z dist init r=2, world=4
2022-05-18T03:35:14.2503077Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgpbeinzf
2022-05-18T03:35:14.2504385Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgpbeinzf/_remote_module_non_scriptable.py
2022-05-18T03:35:14.2543125Z dist init r=3, world=4
2022-05-18T03:35:14.2664244Z dist init r=1, world=4
2022-05-18T03:35:14.2951528Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9r12fmg8
2022-05-18T03:35:14.2952465Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9r12fmg8/_remote_module_non_scriptable.py
2022-05-18T03:35:14.3111264Z dist init r=0, world=4
2022-05-18T03:35:14.3412820Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:35:14.3514262Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:35:14.3616219Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:35:14.3617246Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:14.3617932Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:35:14.3618778Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:14.3619540Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:14.3620340Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:14.3624540Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:35:14.3625112Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:35:14.3625646Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:35:14.3626223Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:35:14.5347398Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:35:14.5391133Z   test_delayed_reduce_scatter_offload_true_prefetch_post_no_shard (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5301
2022-05-18T03:35:14.5424076Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5302
2022-05-18T03:35:14.5451852Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5303
2022-05-18T03:35:14.5480048Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5304
2022-05-18T03:35:15.1406132Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy1v0wcx1
2022-05-18T03:35:15.1406945Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy1v0wcx1/_remote_module_non_scriptable.py
2022-05-18T03:35:15.1443456Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpam5p6g_a
2022-05-18T03:35:15.1444672Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpam5p6g_a/_remote_module_non_scriptable.py
2022-05-18T03:35:15.1562311Z dist init r=2, world=4
2022-05-18T03:35:15.1600417Z dist init r=0, world=4
2022-05-18T03:35:15.1678012Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcdw04pur
2022-05-18T03:35:15.1679644Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcdw04pur/_remote_module_non_scriptable.py
2022-05-18T03:35:15.1719243Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj5gcixk7
2022-05-18T03:35:15.1721111Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj5gcixk7/_remote_module_non_scriptable.py
2022-05-18T03:35:15.1837452Z dist init r=3, world=4
2022-05-18T03:35:15.1876993Z dist init r=1, world=4
2022-05-18T03:35:15.2187768Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:35:15.2288383Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:35:15.2390667Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:35:15.2392326Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:35:15.2393368Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:15.2394071Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:15.2394746Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:15.2395326Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:15.2398709Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:35:15.2399261Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:35:15.2399772Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:35:15.2400316Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:35:15.4507348Z skip: Need at least 2 CUDA devices (0.916s)
2022-05-18T03:35:15.4548748Z   test_delayed_reduce_scatter_offload_true_prefetch_post_none (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5357
2022-05-18T03:35:15.4575226Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5358
2022-05-18T03:35:15.4599943Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5359
2022-05-18T03:35:15.4624317Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5360
2022-05-18T03:35:16.0555553Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphtqp1gbc
2022-05-18T03:35:16.0556458Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphtqp1gbc/_remote_module_non_scriptable.py
2022-05-18T03:35:16.0661829Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp77fnxwir
2022-05-18T03:35:16.0664310Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp77fnxwir/_remote_module_non_scriptable.py
2022-05-18T03:35:16.0716671Z dist init r=2, world=4
2022-05-18T03:35:16.0819283Z dist init r=0, world=4
2022-05-18T03:35:16.0855841Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmputowttfa
2022-05-18T03:35:16.0857800Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmputowttfa/_remote_module_non_scriptable.py
2022-05-18T03:35:16.0991076Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjpth91nu
2022-05-18T03:35:16.0992538Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjpth91nu/_remote_module_non_scriptable.py
2022-05-18T03:35:16.1013610Z dist init r=1, world=4
2022-05-18T03:35:16.1145749Z dist init r=3, world=4
2022-05-18T03:35:16.1328644Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:35:16.1329061Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:35:16.1430071Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:35:16.1431438Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:16.1431974Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:35:16.1432942Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:16.1433616Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:16.1434315Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:16.1438120Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:35:16.1438975Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:35:16.1439524Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:35:16.1441427Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:35:16.3650576Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:35:16.3693821Z   test_delayed_reduce_scatter_offload_true_prefetch_post_shard_grad_op (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5413
2022-05-18T03:35:16.3720964Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5414
2022-05-18T03:35:16.3743557Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5415
2022-05-18T03:35:16.3767145Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5416
2022-05-18T03:35:16.9431249Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpin76baii
2022-05-18T03:35:16.9432434Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpin76baii/_remote_module_non_scriptable.py
2022-05-18T03:35:16.9592381Z dist init r=0, world=4
2022-05-18T03:35:16.9812000Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr7lab4qv
2022-05-18T03:35:16.9813916Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr7lab4qv/_remote_module_non_scriptable.py
2022-05-18T03:35:16.9971258Z dist init r=2, world=4
2022-05-18T03:35:17.0009967Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmn7x52_z
2022-05-18T03:35:17.0012018Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmn7x52_z/_remote_module_non_scriptable.py
2022-05-18T03:35:17.0146721Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3lzyxjfi
2022-05-18T03:35:17.0148430Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3lzyxjfi/_remote_module_non_scriptable.py
2022-05-18T03:35:17.0167748Z dist init r=3, world=4
2022-05-18T03:35:17.0300574Z dist init r=1, world=4
2022-05-18T03:35:17.0707687Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:35:17.0808196Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:35:17.0910381Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:35:17.0911065Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:35:17.0911975Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:17.0912644Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:17.0913265Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:17.0913786Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:17.1017803Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:35:17.1018683Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:35:17.1019292Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:35:17.1019874Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:35:17.2793836Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:35:17.2836935Z   test_delayed_reduce_scatter_offload_true_prefetch_pre_no_shard (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5469
2022-05-18T03:35:17.2863165Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5470
2022-05-18T03:35:17.2887078Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5471
2022-05-18T03:35:17.2911613Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5472
2022-05-18T03:35:17.8556676Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbldr8s05
2022-05-18T03:35:17.8557930Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbldr8s05/_remote_module_non_scriptable.py
2022-05-18T03:35:17.8595364Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7ol8y232
2022-05-18T03:35:17.8596973Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7ol8y232/_remote_module_non_scriptable.py
2022-05-18T03:35:17.8716500Z dist init r=2, world=4
2022-05-18T03:35:17.8753018Z dist init r=1, world=4
2022-05-18T03:35:17.8842732Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf5db_bql
2022-05-18T03:35:17.8845460Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf5db_bql/_remote_module_non_scriptable.py
2022-05-18T03:35:17.8999020Z dist init r=3, world=4
2022-05-18T03:35:17.9319830Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp1a5eson
2022-05-18T03:35:17.9321635Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp1a5eson/_remote_module_non_scriptable.py
2022-05-18T03:35:17.9475703Z dist init r=0, world=4
2022-05-18T03:35:17.9712224Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:35:17.9814530Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:35:17.9815540Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:17.9816109Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:35:17.9816702Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:35:17.9817502Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:17.9818359Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:17.9818994Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:17.9824099Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:35:17.9824579Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:35:17.9824937Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:35:17.9825269Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:35:18.1937787Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:35:18.1980780Z   test_delayed_reduce_scatter_offload_true_prefetch_pre_none (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5525
2022-05-18T03:35:18.2005559Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5526
2022-05-18T03:35:18.2028756Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5527
2022-05-18T03:35:18.2053072Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5528
2022-05-18T03:35:18.8056537Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpiru00bpp
2022-05-18T03:35:18.8057519Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpiru00bpp/_remote_module_non_scriptable.py
2022-05-18T03:35:18.8160655Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyrg0gwnt
2022-05-18T03:35:18.8161532Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyrg0gwnt/_remote_module_non_scriptable.py
2022-05-18T03:35:18.8219753Z dist init r=1, world=4
2022-05-18T03:35:18.8261085Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq7owssbw
2022-05-18T03:35:18.8262714Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq7owssbw/_remote_module_non_scriptable.py
2022-05-18T03:35:18.8299144Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk3wj5fiv
2022-05-18T03:35:18.8301155Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk3wj5fiv/_remote_module_non_scriptable.py
2022-05-18T03:35:18.8322471Z dist init r=0, world=4
2022-05-18T03:35:18.8425086Z dist init r=3, world=4
2022-05-18T03:35:18.8458193Z dist init r=2, world=4
2022-05-18T03:35:18.8731492Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:35:18.8833035Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:35:18.8934703Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:35:18.8935780Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:18.8936529Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:35:18.8937453Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:18.8938195Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:18.8938951Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:18.8942840Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:35:18.8943606Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:35:18.8943970Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:35:18.8946377Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:35:19.1079698Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:35:19.1122105Z   test_delayed_reduce_scatter_offload_true_prefetch_pre_shard_grad_op (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5581
2022-05-18T03:35:19.1148435Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5582
2022-05-18T03:35:19.1172056Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5583
2022-05-18T03:35:19.1196625Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5584
2022-05-18T03:35:19.7430496Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpes2_t6q8
2022-05-18T03:35:19.7431907Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpes2_t6q8/_remote_module_non_scriptable.py
2022-05-18T03:35:19.7528823Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp586ofnyy
2022-05-18T03:35:19.7530130Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp586ofnyy/_remote_module_non_scriptable.py
2022-05-18T03:35:19.7592033Z dist init r=0, world=4
2022-05-18T03:35:19.7612291Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4hkotnzb
2022-05-18T03:35:19.7614171Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4hkotnzb/_remote_module_non_scriptable.py
2022-05-18T03:35:19.7689283Z dist init r=2, world=4
2022-05-18T03:35:19.7774648Z dist init r=1, world=4
2022-05-18T03:35:19.7797782Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjp59ueyr
2022-05-18T03:35:19.7799999Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjp59ueyr/_remote_module_non_scriptable.py
2022-05-18T03:35:19.7953421Z dist init r=3, world=4
2022-05-18T03:35:19.8061349Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:35:19.8100943Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:35:19.8203090Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:35:19.8203701Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:35:19.8204741Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:19.8205285Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:19.8205807Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:19.8265716Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:19.8311592Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:35:19.8312291Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:35:19.8312820Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:35:19.8313355Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:35:20.0224129Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:35:20.0267202Z   test_mixture_of_experts_offload_false_none_no_shard_clip_norm_type_2_0 (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5637
2022-05-18T03:35:20.0293317Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5638
2022-05-18T03:35:20.0317338Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5639
2022-05-18T03:35:20.0341780Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5640
2022-05-18T03:35:20.5996750Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsqc9zfmr
2022-05-18T03:35:20.5997464Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsqc9zfmr/_remote_module_non_scriptable.py
2022-05-18T03:35:20.6157481Z dist init r=0, world=4
2022-05-18T03:35:20.6426861Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeag9sohj
2022-05-18T03:35:20.6427773Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeag9sohj/_remote_module_non_scriptable.py
2022-05-18T03:35:20.6575960Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxbied36d
2022-05-18T03:35:20.6579394Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxbied36d/_remote_module_non_scriptable.py
2022-05-18T03:35:20.6581129Z dist init r=2, world=4
2022-05-18T03:35:20.6694442Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprl6tksfx
2022-05-18T03:35:20.6695938Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprl6tksfx/_remote_module_non_scriptable.py
2022-05-18T03:35:20.6736830Z dist init r=1, world=4
2022-05-18T03:35:20.6852661Z dist init r=3, world=4
2022-05-18T03:35:20.7071890Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:35:20.7174316Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:35:20.7175171Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:35:20.7176112Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:35:20.7177182Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:20.7178295Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:20.7179391Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:20.7180442Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:20.7182570Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:35:20.7183483Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:35:20.7229131Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:35:20.7229713Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:35:20.9368268Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:35:20.9410539Z   test_mixture_of_experts_offload_false_none_no_shard_clip_norm_type_None (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5693
2022-05-18T03:35:20.9436539Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5694
2022-05-18T03:35:20.9460255Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5695
2022-05-18T03:35:20.9484779Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5696
2022-05-18T03:35:21.5161520Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsvkmbhtv
2022-05-18T03:35:21.5162314Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsvkmbhtv/_remote_module_non_scriptable.py
2022-05-18T03:35:21.5170334Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcjgai147
2022-05-18T03:35:21.5172965Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcjgai147/_remote_module_non_scriptable.py
2022-05-18T03:35:21.5192226Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppahwsv26
2022-05-18T03:35:21.5194358Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppahwsv26/_remote_module_non_scriptable.py
2022-05-18T03:35:21.5322920Z dist init r=1, world=4
2022-05-18T03:35:21.5330564Z dist init r=3, world=4
2022-05-18T03:35:21.5351849Z dist init r=0, world=4
2022-05-18T03:35:21.5732830Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps6koe860
2022-05-18T03:35:21.5734458Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps6koe860/_remote_module_non_scriptable.py
2022-05-18T03:35:21.5888550Z dist init r=2, world=4
2022-05-18T03:35:21.6136447Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:35:21.6237759Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:35:21.6338984Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:35:21.6341079Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:35:21.6342042Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:21.6342797Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:21.6343724Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:21.6344538Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:21.6347470Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:35:21.6348155Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:35:21.6348519Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:35:21.6350624Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:35:21.8511117Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:35:21.8555153Z   test_mixture_of_experts_offload_false_none_none_clip_norm_type_2_0 (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5749
2022-05-18T03:35:21.8581536Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5750
2022-05-18T03:35:21.8605804Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5751
2022-05-18T03:35:21.8630004Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5752
2022-05-18T03:35:22.4270506Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptlusk79l
2022-05-18T03:35:22.4271483Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptlusk79l/_remote_module_non_scriptable.py
2022-05-18T03:35:22.4429634Z dist init r=1, world=4
2022-05-18T03:35:22.4633441Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe3ggdc9o
2022-05-18T03:35:22.4634682Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe3ggdc9o/_remote_module_non_scriptable.py
2022-05-18T03:35:22.4790025Z dist init r=3, world=4
2022-05-18T03:35:22.4871483Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk52cj1e0
2022-05-18T03:35:22.4873566Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk52cj1e0/_remote_module_non_scriptable.py
2022-05-18T03:35:22.4990230Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8mpwg2z6
2022-05-18T03:35:22.4992659Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8mpwg2z6/_remote_module_non_scriptable.py
2022-05-18T03:35:22.5031004Z dist init r=2, world=4
2022-05-18T03:35:22.5149329Z dist init r=0, world=4
2022-05-18T03:35:22.5561245Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:35:22.5661767Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:35:22.5763426Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:35:22.5764092Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:35:22.5765001Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:22.5765758Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:22.5766430Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:22.5767123Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:22.5772200Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:35:22.5773184Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:35:22.5773779Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:35:22.5774290Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:35:22.7656721Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:35:22.7701472Z   test_mixture_of_experts_offload_false_none_none_clip_norm_type_None (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5805
2022-05-18T03:35:22.7727571Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5806
2022-05-18T03:35:22.7750421Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5807
2022-05-18T03:35:22.7774303Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5808
2022-05-18T03:35:23.3421992Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9lv1mulc
2022-05-18T03:35:23.3423117Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9lv1mulc/_remote_module_non_scriptable.py
2022-05-18T03:35:23.3579868Z dist init r=1, world=4
2022-05-18T03:35:23.3917714Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp17b_t9so
2022-05-18T03:35:23.3919004Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp17b_t9so/_remote_module_non_scriptable.py
2022-05-18T03:35:23.3935650Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppe5s45w8
2022-05-18T03:35:23.3938128Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppe5s45w8/_remote_module_non_scriptable.py
2022-05-18T03:35:23.4048252Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf7dc8wdc
2022-05-18T03:35:23.4050194Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf7dc8wdc/_remote_module_non_scriptable.py
2022-05-18T03:35:23.4077991Z dist init r=3, world=4
2022-05-18T03:35:23.4097143Z dist init r=2, world=4
2022-05-18T03:35:23.4205842Z dist init r=0, world=4
2022-05-18T03:35:23.4616678Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:35:23.4718713Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:35:23.4719195Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:35:23.4720026Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:23.4720509Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:35:23.4721069Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:23.4721612Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:23.4820610Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:23.4926611Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:35:23.4927048Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:35:23.4927565Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:35:23.4929186Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:35:23.6800563Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:35:23.6843616Z   test_mixture_of_experts_offload_false_none_shard_grad_op_clip_norm_type_2_0 (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5861
2022-05-18T03:35:23.6870242Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5862
2022-05-18T03:35:23.6894217Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5863
2022-05-18T03:35:23.6918452Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5864
2022-05-18T03:35:24.2970206Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzvuq_8w7
2022-05-18T03:35:24.2971189Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzvuq_8w7/_remote_module_non_scriptable.py
2022-05-18T03:35:24.3091927Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpinp_2k4r
2022-05-18T03:35:24.3093119Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpinp_2k4r/_remote_module_non_scriptable.py
2022-05-18T03:35:24.3134167Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnu0p_paf
2022-05-18T03:35:24.3135040Z dist init r=1, world=4
2022-05-18T03:35:24.3135874Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnu0p_paf/_remote_module_non_scriptable.py
2022-05-18T03:35:24.3173828Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx2p1crs2
2022-05-18T03:35:24.3175631Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx2p1crs2/_remote_module_non_scriptable.py
2022-05-18T03:35:24.3256709Z dist init r=3, world=4
2022-05-18T03:35:24.3298380Z dist init r=0, world=4
2022-05-18T03:35:24.3333497Z dist init r=2, world=4
2022-05-18T03:35:24.3568743Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:35:24.3771047Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:35:24.3872477Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:35:24.3873699Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:24.3874475Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:35:24.3875283Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:24.3876009Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:24.3876600Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:24.3880582Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:35:24.3881169Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:35:24.3881707Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:35:24.3882716Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:35:24.5945333Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:35:24.5988395Z   test_mixture_of_experts_offload_false_none_shard_grad_op_clip_norm_type_None (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5917
2022-05-18T03:35:24.6014728Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5918
2022-05-18T03:35:24.6038935Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5919
2022-05-18T03:35:24.6063552Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5920
2022-05-18T03:35:25.2022790Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8i78ykbe
2022-05-18T03:35:25.2023897Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8i78ykbe/_remote_module_non_scriptable.py
2022-05-18T03:35:25.2140066Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3dgwi06g
2022-05-18T03:35:25.2141349Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3dgwi06g/_remote_module_non_scriptable.py
2022-05-18T03:35:25.2183171Z dist init r=2, world=4
2022-05-18T03:35:25.2299814Z dist init r=3, world=4
2022-05-18T03:35:25.2351142Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplyxy2l4e
2022-05-18T03:35:25.2353455Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplyxy2l4e/_remote_module_non_scriptable.py
2022-05-18T03:35:25.2442163Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpukgky96b
2022-05-18T03:35:25.2444452Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpukgky96b/_remote_module_non_scriptable.py
2022-05-18T03:35:25.2511416Z dist init r=1, world=4
2022-05-18T03:35:25.2602288Z dist init r=0, world=4
2022-05-18T03:35:25.2924692Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:35:25.3025642Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:35:25.3026146Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:35:25.3026706Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:35:25.3027554Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:25.3028114Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:25.3028696Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:25.3031047Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:25.3037300Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:35:25.3037871Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:35:25.3038408Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:35:25.3038945Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:35:25.5089787Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:35:25.5133249Z   test_mixture_of_experts_offload_false_prefetch_post_no_shard_clip_norm_type_2_0 (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5973
2022-05-18T03:35:25.5159556Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5974
2022-05-18T03:35:25.5182801Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5975
2022-05-18T03:35:25.5207123Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5976
2022-05-18T03:35:26.1098230Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpska3yix3
2022-05-18T03:35:26.1099068Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpska3yix3/_remote_module_non_scriptable.py
2022-05-18T03:35:26.1112404Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpoq9fmn6g
2022-05-18T03:35:26.1114449Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpoq9fmn6g/_remote_module_non_scriptable.py
2022-05-18T03:35:26.1262516Z dist init r=3, world=4
2022-05-18T03:35:26.1270935Z dist init r=0, world=4
2022-05-18T03:35:26.1484770Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxche9916
2022-05-18T03:35:26.1485969Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxche9916/_remote_module_non_scriptable.py
2022-05-18T03:35:26.1611761Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphma7pljo
2022-05-18T03:35:26.1613944Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphma7pljo/_remote_module_non_scriptable.py
2022-05-18T03:35:26.1643116Z dist init r=2, world=4
2022-05-18T03:35:26.1769180Z dist init r=1, world=4
2022-05-18T03:35:26.2054200Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:35:26.2084403Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:35:26.2085148Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:35:26.2086152Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:26.2086687Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:35:26.2087187Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:26.2087709Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:26.2156192Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:26.2192479Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:35:26.2193012Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:35:26.2193555Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:35:26.2194075Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:35:26.4234738Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:35:26.4278828Z   test_mixture_of_experts_offload_false_prefetch_post_no_shard_clip_norm_type_None (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6029
2022-05-18T03:35:26.4305168Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6030
2022-05-18T03:35:26.4328923Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6031
2022-05-18T03:35:26.4353649Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6032
2022-05-18T03:35:27.0017458Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp13yc08tp
2022-05-18T03:35:27.0018303Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp13yc08tp/_remote_module_non_scriptable.py
2022-05-18T03:35:27.0103895Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8mgvr2ss
2022-05-18T03:35:27.0104679Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8mgvr2ss/_remote_module_non_scriptable.py
2022-05-18T03:35:27.0178888Z dist init r=2, world=4
2022-05-18T03:35:27.0262071Z dist init r=3, world=4
2022-05-18T03:35:27.0525711Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8tye7ojb
2022-05-18T03:35:27.0526342Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8tye7ojb/_remote_module_non_scriptable.py
2022-05-18T03:35:27.0659399Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsf3lkoz1
2022-05-18T03:35:27.0660647Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsf3lkoz1/_remote_module_non_scriptable.py
2022-05-18T03:35:27.0683398Z dist init r=0, world=4
2022-05-18T03:35:27.0816854Z dist init r=1, world=4
2022-05-18T03:35:27.0991650Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:35:27.1091868Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:35:27.1193942Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:35:27.1194850Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:35:27.1196044Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:27.1196664Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:27.1197173Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:27.1197694Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:27.1301878Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:35:27.1302453Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:35:27.1303154Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:35:27.1303702Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:35:27.3380081Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:35:27.3422724Z   test_mixture_of_experts_offload_false_prefetch_post_none_clip_norm_type_2_0 (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6085
2022-05-18T03:35:27.3449149Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6086
2022-05-18T03:35:27.3472676Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6087
2022-05-18T03:35:27.3496518Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6088
2022-05-18T03:35:27.9306788Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_e46vejk
2022-05-18T03:35:27.9307829Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_e46vejk/_remote_module_non_scriptable.py
2022-05-18T03:35:27.9470029Z dist init r=0, world=4
2022-05-18T03:35:27.9750192Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5jukbivy
2022-05-18T03:35:27.9750958Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjoz2wsq4
2022-05-18T03:35:27.9751652Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5jukbivy/_remote_module_non_scriptable.py
2022-05-18T03:35:27.9752065Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjoz2wsq4/_remote_module_non_scriptable.py
2022-05-18T03:35:27.9820403Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqvldlv7j
2022-05-18T03:35:27.9822076Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqvldlv7j/_remote_module_non_scriptable.py
2022-05-18T03:35:27.9910722Z dist init r=3, world=4
2022-05-18T03:35:27.9910969Z dist init r=2, world=4
2022-05-18T03:35:27.9979927Z dist init r=1, world=4
2022-05-18T03:35:28.0291748Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:35:28.0292367Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:35:28.0292952Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:35:28.0293616Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:28.0294020Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:35:28.0294566Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:28.0295131Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:28.0295868Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:28.0402204Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:35:28.0402781Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:35:28.0403358Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:35:28.0405577Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:35:28.2522960Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:35:28.2566154Z   test_mixture_of_experts_offload_false_prefetch_post_none_clip_norm_type_None (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6141
2022-05-18T03:35:28.2592655Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6142
2022-05-18T03:35:28.2616635Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6143
2022-05-18T03:35:28.2641461Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6144
2022-05-18T03:35:28.8837123Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5pq2oloa
2022-05-18T03:35:28.8838275Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5pq2oloa/_remote_module_non_scriptable.py
2022-05-18T03:35:28.8877514Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp176r0he1
2022-05-18T03:35:28.8879138Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp176r0he1/_remote_module_non_scriptable.py
2022-05-18T03:35:28.8998100Z dist init r=3, world=4
2022-05-18T03:35:28.9037554Z dist init r=2, world=4
2022-05-18T03:35:28.9375233Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptsop9uoi
2022-05-18T03:35:28.9376677Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptsop9uoi/_remote_module_non_scriptable.py
2022-05-18T03:35:28.9531999Z dist init r=1, world=4
2022-05-18T03:35:28.9858127Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3eh3120v
2022-05-18T03:35:28.9858785Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3eh3120v/_remote_module_non_scriptable.py
2022-05-18T03:35:29.0017194Z dist init r=0, world=4
2022-05-18T03:35:29.0428445Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:35:29.0530880Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:35:29.0531395Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:35:29.0531821Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:35:29.0532463Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:29.0533016Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:29.0533537Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:29.0632338Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:29.0639484Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:35:29.0642538Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:35:29.0643078Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:35:29.0643629Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:35:29.2669276Z skip: Need at least 2 CUDA devices (1.014s)
2022-05-18T03:35:29.2711796Z   test_mixture_of_experts_offload_false_prefetch_post_shard_grad_op_clip_norm_type_2_0 (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6197
2022-05-18T03:35:29.2737377Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6198
2022-05-18T03:35:29.2761263Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6199
2022-05-18T03:35:29.2785404Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6200
2022-05-18T03:35:29.8963088Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzpirimuq
2022-05-18T03:35:29.8964511Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzpirimuq/_remote_module_non_scriptable.py
2022-05-18T03:35:29.8969363Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpln6da0yw
2022-05-18T03:35:29.8971891Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpln6da0yw/_remote_module_non_scriptable.py
2022-05-18T03:35:29.9023114Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa0c_fpup
2022-05-18T03:35:29.9025343Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa0c_fpup/_remote_module_non_scriptable.py
2022-05-18T03:35:29.9127297Z dist init r=2, world=4
2022-05-18T03:35:29.9127647Z dist init r=3, world=4
2022-05-18T03:35:29.9166231Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9yed1cbp
2022-05-18T03:35:29.9168068Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9yed1cbp/_remote_module_non_scriptable.py
2022-05-18T03:35:29.9184025Z dist init r=0, world=4
2022-05-18T03:35:29.9320502Z dist init r=1, world=4
2022-05-18T03:35:29.9695777Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:35:29.9796689Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:35:29.9899625Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:35:29.9900313Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:35:29.9901158Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:29.9901858Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:29.9902558Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:29.9903293Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:29.9906802Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:35:29.9907685Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:35:29.9908546Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:35:29.9909429Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:35:30.1811738Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:35:30.1854639Z   test_mixture_of_experts_offload_false_prefetch_post_shard_grad_op_clip_norm_type_None (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6253
2022-05-18T03:35:30.1881258Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6254
2022-05-18T03:35:30.1905119Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6255
2022-05-18T03:35:30.1929312Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6256
2022-05-18T03:35:30.8541208Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4tqs3p3p
2022-05-18T03:35:30.8542603Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4tqs3p3p/_remote_module_non_scriptable.py
2022-05-18T03:35:30.8699152Z dist init r=3, world=4
2022-05-18T03:35:30.8779218Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps8e3wsju
2022-05-18T03:35:30.8780799Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps8e3wsju/_remote_module_non_scriptable.py
2022-05-18T03:35:30.8929047Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnmityk_7
2022-05-18T03:35:30.8930398Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnmityk_7/_remote_module_non_scriptable.py
2022-05-18T03:35:30.8936488Z dist init r=2, world=4
2022-05-18T03:35:30.8937012Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkxon27i8
2022-05-18T03:35:30.8939015Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkxon27i8/_remote_module_non_scriptable.py
2022-05-18T03:35:30.9085468Z dist init r=1, world=4
2022-05-18T03:35:30.9094472Z dist init r=0, world=4
2022-05-18T03:35:30.9496278Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:35:30.9597046Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:35:30.9699947Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:35:30.9701089Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:30.9701521Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:35:30.9702004Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:30.9702520Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:30.9800288Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:30.9807113Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:35:30.9807674Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:35:30.9808207Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:35:30.9811113Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:35:31.1957418Z skip: Need at least 2 CUDA devices (1.014s)
2022-05-18T03:35:31.2001122Z   test_mixture_of_experts_offload_false_prefetch_pre_no_shard_clip_norm_type_2_0 (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6309
2022-05-18T03:35:31.2026887Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6310
2022-05-18T03:35:31.2050451Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6311
2022-05-18T03:35:31.2074761Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6312
2022-05-18T03:35:31.8023928Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppt7juc1d
2022-05-18T03:35:31.8024895Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppt7juc1d/_remote_module_non_scriptable.py
2022-05-18T03:35:31.8132609Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp4wl35_u
2022-05-18T03:35:31.8133432Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp4wl35_u/_remote_module_non_scriptable.py
2022-05-18T03:35:31.8182505Z dist init r=3, world=4
2022-05-18T03:35:31.8289854Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4476zwft
2022-05-18T03:35:31.8290801Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4476zwft/_remote_module_non_scriptable.py
2022-05-18T03:35:31.8291356Z dist init r=0, world=4
2022-05-18T03:35:31.8302998Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx8sq_g51
2022-05-18T03:35:31.8305141Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx8sq_g51/_remote_module_non_scriptable.py
2022-05-18T03:35:31.8445588Z dist init r=2, world=4
2022-05-18T03:35:31.8459211Z dist init r=1, world=4
2022-05-18T03:35:31.8653463Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:35:31.8693351Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:35:31.8795435Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:35:31.8796089Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:31.8796495Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:35:31.8797002Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:31.8797524Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:31.8857359Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:31.8903219Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:35:31.8903769Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:35:31.8904315Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:35:31.8904881Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:35:32.1102078Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:35:32.1146750Z   test_mixture_of_experts_offload_false_prefetch_pre_no_shard_clip_norm_type_None (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6365
2022-05-18T03:35:32.1172752Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6366
2022-05-18T03:35:32.1195711Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6367
2022-05-18T03:35:32.1219509Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6368
2022-05-18T03:35:32.6877071Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2w_n9qkg
2022-05-18T03:35:32.6877840Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjwdvdc7g
2022-05-18T03:35:32.6878559Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2w_n9qkg/_remote_module_non_scriptable.py
2022-05-18T03:35:32.6879256Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjwdvdc7g/_remote_module_non_scriptable.py
2022-05-18T03:35:32.6903855Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkvrzkled
2022-05-18T03:35:32.6905942Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkvrzkled/_remote_module_non_scriptable.py
2022-05-18T03:35:32.7040348Z dist init r=1, world=4
2022-05-18T03:35:32.7040669Z dist init r=2, world=4
2022-05-18T03:35:32.7068107Z dist init r=0, world=4
2022-05-18T03:35:32.7503003Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3ldqi_4b
2022-05-18T03:35:32.7503622Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3ldqi_4b/_remote_module_non_scriptable.py
2022-05-18T03:35:32.7659643Z dist init r=3, world=4
2022-05-18T03:35:32.7782391Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:35:32.7884640Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:35:32.7985488Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:35:32.7986096Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:35:32.7986998Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:32.7987836Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:32.7988539Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:32.7989307Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:32.7994433Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:35:32.7995047Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:35:32.7995674Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:35:32.7996220Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:35:33.0247177Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:35:33.0307088Z   test_mixture_of_experts_offload_false_prefetch_pre_none_clip_norm_type_2_0 (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6421
2022-05-18T03:35:33.0333442Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6422
2022-05-18T03:35:33.0358061Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6423
2022-05-18T03:35:33.0389479Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6424
2022-05-18T03:35:33.6451843Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe79agzre
2022-05-18T03:35:33.6452656Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe79agzre/_remote_module_non_scriptable.py
2022-05-18T03:35:33.6597348Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc3crs7et
2022-05-18T03:35:33.6599444Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc3crs7et/_remote_module_non_scriptable.py
2022-05-18T03:35:33.6617466Z dist init r=2, world=4
2022-05-18T03:35:33.6627367Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3lhcdc7o
2022-05-18T03:35:33.6629604Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3lhcdc7o/_remote_module_non_scriptable.py
2022-05-18T03:35:33.6638480Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp08l1y06x
2022-05-18T03:35:33.6640554Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp08l1y06x/_remote_module_non_scriptable.py
2022-05-18T03:35:33.6764701Z dist init r=3, world=4
2022-05-18T03:35:33.6789856Z dist init r=0, world=4
2022-05-18T03:35:33.6801432Z dist init r=1, world=4
2022-05-18T03:35:33.6974006Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:35:33.7074811Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:35:33.7177280Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:35:33.7177995Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:35:33.7178613Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:33.7179139Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:33.7179812Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:33.7180391Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:33.7284650Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:35:33.7285036Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:35:33.7285551Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:35:33.7288124Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:35:33.9415778Z skip: Need at least 2 CUDA devices (0.917s)
2022-05-18T03:35:33.9459559Z   test_mixture_of_experts_offload_false_prefetch_pre_none_clip_norm_type_None (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6477
2022-05-18T03:35:33.9486556Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6478
2022-05-18T03:35:33.9510831Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6479
2022-05-18T03:35:33.9534692Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6480
2022-05-18T03:35:34.5385627Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmxft6hvw
2022-05-18T03:35:34.5386994Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmxft6hvw/_remote_module_non_scriptable.py
2022-05-18T03:35:34.5547865Z dist init r=3, world=4
2022-05-18T03:35:34.5599368Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw50hlzl4
2022-05-18T03:35:34.5601244Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw50hlzl4/_remote_module_non_scriptable.py
2022-05-18T03:35:34.5639001Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcxgoevgi
2022-05-18T03:35:34.5640411Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcxgoevgi/_remote_module_non_scriptable.py
2022-05-18T03:35:34.5706627Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0q86g9ok
2022-05-18T03:35:34.5708582Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0q86g9ok/_remote_module_non_scriptable.py
2022-05-18T03:35:34.5762787Z dist init r=0, world=4
2022-05-18T03:35:34.5798888Z dist init r=2, world=4
2022-05-18T03:35:34.5867243Z dist init r=1, world=4
2022-05-18T03:35:34.6160170Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:35:34.6362786Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:35:34.6464383Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:35:34.6465103Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:35:34.6465938Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:34.6466478Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:34.6467002Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:34.6467520Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:34.6571849Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:35:34.6572555Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:35:34.6573026Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:35:34.6573593Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:35:34.8561461Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:35:34.8606907Z   test_mixture_of_experts_offload_false_prefetch_pre_shard_grad_op_clip_norm_type_2_0 (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6533
2022-05-18T03:35:34.8633092Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6534
2022-05-18T03:35:34.8656506Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6535
2022-05-18T03:35:34.8680583Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6536
2022-05-18T03:35:35.4289754Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyy511p0h
2022-05-18T03:35:35.4290517Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyy511p0h/_remote_module_non_scriptable.py
2022-05-18T03:35:35.4346232Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp84f52pj5
2022-05-18T03:35:35.4347434Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp84f52pj5/_remote_module_non_scriptable.py
2022-05-18T03:35:35.4348094Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdmvs_9sq
2022-05-18T03:35:35.4350351Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdmvs_9sq/_remote_module_non_scriptable.py
2022-05-18T03:35:35.4362960Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpot73wgc9
2022-05-18T03:35:35.4365114Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpot73wgc9/_remote_module_non_scriptable.py
2022-05-18T03:35:35.4460052Z dist init r=2, world=4
2022-05-18T03:35:35.4510171Z dist init r=3, world=4
2022-05-18T03:35:35.4512324Z dist init r=1, world=4
2022-05-18T03:35:35.4526740Z dist init r=0, world=4
2022-05-18T03:35:35.4822532Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:35:35.5025159Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:35:35.5126791Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:35:35.5127198Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:35:35.5127825Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:35.5128344Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:35.5128864Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:35.5227444Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:35.5234695Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:35:35.5235537Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:35:35.5235923Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:35:35.5236439Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:35:35.6705438Z skip: Need at least 2 CUDA devices (0.814s)
2022-05-18T03:35:35.6748784Z   test_mixture_of_experts_offload_false_prefetch_pre_shard_grad_op_clip_norm_type_None (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6589
2022-05-18T03:35:35.6774858Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6590
2022-05-18T03:35:35.6799415Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6591
2022-05-18T03:35:35.6824383Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6592
2022-05-18T03:35:36.2897617Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps4045wp0
2022-05-18T03:35:36.2898540Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps4045wp0/_remote_module_non_scriptable.py
2022-05-18T03:35:36.2983289Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp004y4e4b
2022-05-18T03:35:36.2984534Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp004y4e4b/_remote_module_non_scriptable.py
2022-05-18T03:35:36.3062560Z dist init r=0, world=4
2022-05-18T03:35:36.3142286Z dist init r=2, world=4
2022-05-18T03:35:36.3170883Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzr8mhw1h
2022-05-18T03:35:36.3172489Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzr8mhw1h/_remote_module_non_scriptable.py
2022-05-18T03:35:36.3264879Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy76d80j9
2022-05-18T03:35:36.3267174Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy76d80j9/_remote_module_non_scriptable.py
2022-05-18T03:35:36.3334426Z dist init r=3, world=4
2022-05-18T03:35:36.3431238Z dist init r=1, world=4
2022-05-18T03:35:36.3553153Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:35:36.3653468Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:35:36.3747431Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:35:36.3747837Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:35:36.3748451Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:36.3749064Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:36.3756776Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:36.3757305Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:36.3863378Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:35:36.3864078Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:35:36.3864539Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:35:36.3865165Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:35:36.5850449Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:35:36.5895754Z   test_mixture_of_experts_offload_true_none_no_shard_clip_norm_type_2_0 (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6645
2022-05-18T03:35:36.5922186Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6646
2022-05-18T03:35:36.5945941Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6647
2022-05-18T03:35:36.5970604Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6648
2022-05-18T03:35:37.1832361Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmparye9ieg
2022-05-18T03:35:37.1833376Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmparye9ieg/_remote_module_non_scriptable.py
2022-05-18T03:35:37.1926710Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq8p47xlg
2022-05-18T03:35:37.1928585Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq8p47xlg/_remote_module_non_scriptable.py
2022-05-18T03:35:37.1995350Z dist init r=3, world=4
2022-05-18T03:35:37.2083204Z dist init r=1, world=4
2022-05-18T03:35:37.2159865Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw12zxgmt
2022-05-18T03:35:37.2161529Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptp1g6e1f
2022-05-18T03:35:37.2162191Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw12zxgmt/_remote_module_non_scriptable.py
2022-05-18T03:35:37.2163657Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptp1g6e1f/_remote_module_non_scriptable.py
2022-05-18T03:35:37.2318317Z dist init r=2, world=4
2022-05-18T03:35:37.2318620Z dist init r=0, world=4
2022-05-18T03:35:37.2630454Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:35:37.2731551Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:35:37.2834867Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:35:37.2835852Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:37.2836416Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:35:37.2837123Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:37.2837922Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:37.2838585Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:37.2841946Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:35:37.2842690Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:35:37.2843370Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:35:37.2843987Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:35:37.4997056Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:35:37.5040954Z   test_mixture_of_experts_offload_true_none_no_shard_clip_norm_type_None (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6701
2022-05-18T03:35:37.5067991Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6702
2022-05-18T03:35:37.5092101Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6703
2022-05-18T03:35:37.5116205Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6704
2022-05-18T03:35:38.0973824Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4ilfxn8l
2022-05-18T03:35:38.0975458Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4ilfxn8l/_remote_module_non_scriptable.py
2022-05-18T03:35:38.1079788Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcb8wwwgm
2022-05-18T03:35:38.1081372Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcb8wwwgm/_remote_module_non_scriptable.py
2022-05-18T03:35:38.1137291Z dist init r=0, world=4
2022-05-18T03:35:38.1160135Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk1v7bs2m
2022-05-18T03:35:38.1162006Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk1v7bs2m/_remote_module_non_scriptable.py
2022-05-18T03:35:38.1244245Z dist init r=1, world=4
2022-05-18T03:35:38.1323403Z dist init r=2, world=4
2022-05-18T03:35:38.1405456Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_xb8oaf5
2022-05-18T03:35:38.1407573Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_xb8oaf5/_remote_module_non_scriptable.py
2022-05-18T03:35:38.1563998Z dist init r=3, world=4
2022-05-18T03:35:38.1672573Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:35:38.1735907Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:35:38.1837663Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:35:38.1838369Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:35:38.1839182Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:38.1839710Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:38.1840227Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:38.1876285Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:38.1946108Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:35:38.1946665Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:35:38.1947231Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:35:38.1947761Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:35:38.4143855Z skip: Need at least 2 CUDA devices (0.915s)
2022-05-18T03:35:38.4188255Z   test_mixture_of_experts_offload_true_none_none_clip_norm_type_2_0 (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6757
2022-05-18T03:35:38.4215337Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6758
2022-05-18T03:35:38.4239445Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6759
2022-05-18T03:35:38.4263866Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6760
2022-05-18T03:35:38.9876992Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf7vb0icr
2022-05-18T03:35:38.9877721Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf7vb0icr/_remote_module_non_scriptable.py
2022-05-18T03:35:38.9943897Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvbsmmze9
2022-05-18T03:35:38.9945497Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvbsmmze9/_remote_module_non_scriptable.py
2022-05-18T03:35:39.0035138Z dist init r=2, world=4
2022-05-18T03:35:39.0102056Z dist init r=3, world=4
2022-05-18T03:35:39.0353062Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptji4j74g
2022-05-18T03:35:39.0354833Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptji4j74g/_remote_module_non_scriptable.py
2022-05-18T03:35:39.0391238Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplgyrlyup
2022-05-18T03:35:39.0392989Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplgyrlyup/_remote_module_non_scriptable.py
2022-05-18T03:35:39.0512207Z dist init r=0, world=4
2022-05-18T03:35:39.0549286Z dist init r=1, world=4
2022-05-18T03:35:39.0861086Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:35:39.0861471Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:35:39.0961855Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:35:39.0962545Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:35:39.0963660Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:39.0964857Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:39.0965757Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:39.0966357Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:39.1070159Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:35:39.1070852Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:35:39.1071384Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:35:39.1071917Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:35:39.3289723Z skip: Need at least 2 CUDA devices (0.915s)
2022-05-18T03:35:39.3335173Z   test_mixture_of_experts_offload_true_none_none_clip_norm_type_None (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6813
2022-05-18T03:35:39.3360678Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6814
2022-05-18T03:35:39.3383827Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6815
2022-05-18T03:35:39.3408171Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6816
2022-05-18T03:35:39.9376294Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp35vgqhdr
2022-05-18T03:35:39.9378244Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp35vgqhdr/_remote_module_non_scriptable.py
2022-05-18T03:35:39.9466044Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9qajtf1t
2022-05-18T03:35:39.9467647Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9qajtf1t/_remote_module_non_scriptable.py
2022-05-18T03:35:39.9489140Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqyq8kz9y
2022-05-18T03:35:39.9491522Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqyq8kz9y/_remote_module_non_scriptable.py
2022-05-18T03:35:39.9537027Z dist init r=2, world=4
2022-05-18T03:35:39.9611596Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqoyeb4wj
2022-05-18T03:35:39.9613184Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqoyeb4wj/_remote_module_non_scriptable.py
2022-05-18T03:35:39.9626372Z dist init r=3, world=4
2022-05-18T03:35:39.9649887Z dist init r=0, world=4
2022-05-18T03:35:39.9770170Z dist init r=1, world=4
2022-05-18T03:35:40.0037717Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:35:40.0138765Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:35:40.0241246Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:35:40.0242236Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:40.0242739Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:35:40.0243605Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:40.0244463Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:40.0245258Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:40.0249514Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:35:40.0250242Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:35:40.0251132Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:35:40.0251914Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:35:40.2435353Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:35:40.2478802Z   test_mixture_of_experts_offload_true_none_shard_grad_op_clip_norm_type_2_0 (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6869
2022-05-18T03:35:40.2507377Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6870
2022-05-18T03:35:40.2530635Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6871
2022-05-18T03:35:40.2554997Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6872
2022-05-18T03:35:40.8448029Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp737m6y8i
2022-05-18T03:35:40.8448798Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp737m6y8i/_remote_module_non_scriptable.py
2022-05-18T03:35:40.8561137Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp1d6oz1v
2022-05-18T03:35:40.8561916Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp1d6oz1v/_remote_module_non_scriptable.py
2022-05-18T03:35:40.8606451Z dist init r=0, world=4
2022-05-18T03:35:40.8720184Z dist init r=1, world=4
2022-05-18T03:35:40.8768225Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmezh5zz7
2022-05-18T03:35:40.8770028Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmezh5zz7/_remote_module_non_scriptable.py
2022-05-18T03:35:40.8842492Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpex6td37h
2022-05-18T03:35:40.8844560Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpex6td37h/_remote_module_non_scriptable.py
2022-05-18T03:35:40.8925840Z dist init r=3, world=4
2022-05-18T03:35:40.9000532Z dist init r=2, world=4
2022-05-18T03:35:40.9134974Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:35:40.9232949Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:35:40.9233416Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:35:40.9233793Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:35:40.9234611Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:40.9235230Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:40.9235788Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:40.9236918Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:40.9341758Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:35:40.9342452Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:35:40.9343030Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:35:40.9343383Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:35:41.1581692Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:35:41.1625431Z   test_mixture_of_experts_offload_true_none_shard_grad_op_clip_norm_type_None (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6925
2022-05-18T03:35:41.1651729Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6926
2022-05-18T03:35:41.1675062Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6927
2022-05-18T03:35:41.1700801Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6928
2022-05-18T03:35:41.7725955Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfmxwvqwo
2022-05-18T03:35:41.7726795Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfmxwvqwo/_remote_module_non_scriptable.py
2022-05-18T03:35:41.7832341Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp26nd173o
2022-05-18T03:35:41.7833096Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp26nd173o/_remote_module_non_scriptable.py
2022-05-18T03:35:41.7884996Z dist init r=2, world=4
2022-05-18T03:35:41.7890212Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsnoyer3y
2022-05-18T03:35:41.7892702Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsnoyer3y/_remote_module_non_scriptable.py
2022-05-18T03:35:41.7957589Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbsh24iw7
2022-05-18T03:35:41.7959385Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbsh24iw7/_remote_module_non_scriptable.py
2022-05-18T03:35:41.7995985Z dist init r=3, world=4
2022-05-18T03:35:41.8052843Z dist init r=1, world=4
2022-05-18T03:35:41.8116697Z dist init r=0, world=4
2022-05-18T03:35:41.8464291Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:35:41.8565446Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:35:41.8667875Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:35:41.8668991Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:41.8669789Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:41.8670323Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:35:41.8671971Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:41.8672740Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:41.8676568Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:35:41.8677126Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:35:41.8677672Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:35:41.8680495Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:35:42.0727599Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:35:42.0770149Z   test_mixture_of_experts_offload_true_prefetch_post_no_shard_clip_norm_type_2_0 (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6981
2022-05-18T03:35:42.0798067Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6982
2022-05-18T03:35:42.0828262Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6983
2022-05-18T03:35:42.0856459Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6984
2022-05-18T03:35:42.7430313Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdmg5qks8
2022-05-18T03:35:42.7431012Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5b1xs9gb
2022-05-18T03:35:42.7432105Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdmg5qks8/_remote_module_non_scriptable.py
2022-05-18T03:35:42.7433032Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5b1xs9gb/_remote_module_non_scriptable.py
2022-05-18T03:35:42.7593796Z dist init r=1, world=4
2022-05-18T03:35:42.7596322Z dist init r=0, world=4
2022-05-18T03:35:42.8025321Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa4m_i0s7
2022-05-18T03:35:42.8026072Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa4m_i0s7/_remote_module_non_scriptable.py
2022-05-18T03:35:42.8180014Z dist init r=3, world=4
2022-05-18T03:35:42.8398427Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2t2l47hf
2022-05-18T03:35:42.8400531Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2t2l47hf/_remote_module_non_scriptable.py
2022-05-18T03:35:42.8553606Z dist init r=2, world=4
2022-05-18T03:35:42.8691451Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:35:42.8792560Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:35:42.8892863Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:35:42.8893605Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:35:42.8894525Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:42.8895148Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:42.8896060Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:42.8896579Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:42.9002306Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:35:42.9002964Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:35:42.9003586Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:35:42.9004219Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:35:43.0885295Z skip: Need at least 2 CUDA devices (1.016s)
2022-05-18T03:35:43.0927948Z   test_mixture_of_experts_offload_true_prefetch_post_no_shard_clip_norm_type_None (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7037
2022-05-18T03:35:43.0953397Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7038
2022-05-18T03:35:43.0976860Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7039
2022-05-18T03:35:43.1001236Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7040
2022-05-18T03:35:43.6880824Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxbi8th16
2022-05-18T03:35:43.6881648Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxbi8th16/_remote_module_non_scriptable.py
2022-05-18T03:35:43.7039786Z dist init r=2, world=4
2022-05-18T03:35:43.7168492Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4d0_36zd
2022-05-18T03:35:43.7169677Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4d0_36zd/_remote_module_non_scriptable.py
2022-05-18T03:35:43.7315802Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpilfx959m
2022-05-18T03:35:43.7317322Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpilfx959m/_remote_module_non_scriptable.py
2022-05-18T03:35:43.7326930Z dist init r=0, world=4
2022-05-18T03:35:43.7378414Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbh7m9_1m
2022-05-18T03:35:43.7380550Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbh7m9_1m/_remote_module_non_scriptable.py
2022-05-18T03:35:43.7475592Z dist init r=1, world=4
2022-05-18T03:35:43.7535434Z dist init r=3, world=4
2022-05-18T03:35:43.7644645Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:35:43.7744458Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:35:43.7786550Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:35:43.7787587Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:35:43.7788302Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:43.7788913Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:43.7847064Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:43.7847621Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:43.7953118Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:35:43.7953710Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:35:43.7954260Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:35:43.7954797Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:35:44.0028478Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:35:44.0070429Z   test_mixture_of_experts_offload_true_prefetch_post_none_clip_norm_type_2_0 (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7093
2022-05-18T03:35:44.0096123Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7094
2022-05-18T03:35:44.0119727Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7095
2022-05-18T03:35:44.0143975Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7096
2022-05-18T03:35:44.6113604Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi8_hvu_v
2022-05-18T03:35:44.6114335Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi8_hvu_v/_remote_module_non_scriptable.py
2022-05-18T03:35:44.6248977Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmsn_1uxw
2022-05-18T03:35:44.6249765Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmsn_1uxw/_remote_module_non_scriptable.py
2022-05-18T03:35:44.6272291Z dist init r=2, world=4
2022-05-18T03:35:44.6405854Z dist init r=1, world=4
2022-05-18T03:35:44.6447318Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwg1gfx_y
2022-05-18T03:35:44.6449829Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwg1gfx_y/_remote_module_non_scriptable.py
2022-05-18T03:35:44.6569881Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkw_66kla
2022-05-18T03:35:44.6571492Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkw_66kla/_remote_module_non_scriptable.py
2022-05-18T03:35:44.6605977Z dist init r=3, world=4
2022-05-18T03:35:44.6726924Z dist init r=0, world=4
2022-05-18T03:35:44.6986518Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:35:44.7088275Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:35:44.7089226Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:44.7089808Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:35:44.7090578Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:35:44.7091306Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:44.7092001Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:44.7092637Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:44.7098107Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:35:44.7098705Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:35:44.7099125Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:35:44.7099484Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:35:44.9170665Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:35:44.9213679Z   test_mixture_of_experts_offload_true_prefetch_post_none_clip_norm_type_None (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7149
2022-05-18T03:35:44.9239901Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7150
2022-05-18T03:35:44.9262289Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7151
2022-05-18T03:35:44.9287080Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7152
2022-05-18T03:35:45.5219885Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxyrh2ru4
2022-05-18T03:35:45.5220707Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxyrh2ru4/_remote_module_non_scriptable.py
2022-05-18T03:35:45.5377686Z dist init r=1, world=4
2022-05-18T03:35:45.5528027Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgvfjpb44
2022-05-18T03:35:45.5529043Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgvfjpb44/_remote_module_non_scriptable.py
2022-05-18T03:35:45.5643813Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqp5ymhr6
2022-05-18T03:35:45.5644947Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqp5ymhr6/_remote_module_non_scriptable.py
2022-05-18T03:35:45.5684964Z dist init r=0, world=4
2022-05-18T03:35:45.5800338Z dist init r=2, world=4
2022-05-18T03:35:45.5824806Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfmtumx9l
2022-05-18T03:35:45.5826740Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfmtumx9l/_remote_module_non_scriptable.py
2022-05-18T03:35:45.5981153Z dist init r=3, world=4
2022-05-18T03:35:45.6097048Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:35:45.6097651Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:35:45.6098312Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:35:45.6099161Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:45.6099671Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:35:45.6100223Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:45.6100749Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:45.6101272Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:45.6203617Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:35:45.6204310Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:35:45.6204746Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:35:45.6207761Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:35:45.8313711Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:35:45.8356930Z   test_mixture_of_experts_offload_true_prefetch_post_shard_grad_op_clip_norm_type_2_0 (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7205
2022-05-18T03:35:45.8383627Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7206
2022-05-18T03:35:45.8407248Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7207
2022-05-18T03:35:45.8431297Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7208
2022-05-18T03:35:46.4114043Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj6ol34kn
2022-05-18T03:35:46.4115411Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj6ol34kn/_remote_module_non_scriptable.py
2022-05-18T03:35:46.4173822Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7q0423uf
2022-05-18T03:35:46.4175140Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7q0423uf/_remote_module_non_scriptable.py
2022-05-18T03:35:46.4275125Z dist init r=2, world=4
2022-05-18T03:35:46.4333560Z dist init r=1, world=4
2022-05-18T03:35:46.4653639Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplkmbe3af
2022-05-18T03:35:46.4654706Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplkmbe3af/_remote_module_non_scriptable.py
2022-05-18T03:35:46.4670543Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2925veau
2022-05-18T03:35:46.4672144Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2925veau/_remote_module_non_scriptable.py
2022-05-18T03:35:46.4813712Z dist init r=0, world=4
2022-05-18T03:35:46.4826952Z dist init r=3, world=4
2022-05-18T03:35:46.5087865Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:35:46.5147244Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:35:46.5147879Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:35:46.5149043Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:46.5149790Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:35:46.5150410Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:46.5150941Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:46.5190213Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:46.5255971Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:35:46.5256630Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:35:46.5257269Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:35:46.5257913Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:35:46.7458591Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:35:46.7501172Z   test_mixture_of_experts_offload_true_prefetch_post_shard_grad_op_clip_norm_type_None (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7261
2022-05-18T03:35:46.7527227Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7262
2022-05-18T03:35:46.7550859Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7263
2022-05-18T03:35:46.7574972Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7264
2022-05-18T03:35:47.3335733Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnpxprqfb
2022-05-18T03:35:47.3336854Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnpxprqfb/_remote_module_non_scriptable.py
2022-05-18T03:35:47.3344610Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4vlj1a55
2022-05-18T03:35:47.3346166Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4vlj1a55/_remote_module_non_scriptable.py
2022-05-18T03:35:47.3379019Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp32l_4p7n
2022-05-18T03:35:47.3381214Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp32l_4p7n/_remote_module_non_scriptable.py
2022-05-18T03:35:47.3500850Z dist init r=0, world=4
2022-05-18T03:35:47.3506072Z dist init r=2, world=4
2022-05-18T03:35:47.3538275Z dist init r=1, world=4
2022-05-18T03:35:47.3967081Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_zmmnapz
2022-05-18T03:35:47.3967798Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_zmmnapz/_remote_module_non_scriptable.py
2022-05-18T03:35:47.4122132Z dist init r=3, world=4
2022-05-18T03:35:47.4252088Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:35:47.4252489Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:35:47.4354519Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:35:47.4355294Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:35:47.4356449Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:47.4357024Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:47.4357551Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:47.4358061Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:47.4364699Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:35:47.4365285Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:35:47.4365823Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:35:47.4366379Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:35:47.6601902Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:35:47.6645957Z   test_mixture_of_experts_offload_true_prefetch_pre_no_shard_clip_norm_type_2_0 (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7317
2022-05-18T03:35:47.6672286Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7318
2022-05-18T03:35:47.6695903Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7319
2022-05-18T03:35:47.6721030Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7320
2022-05-18T03:35:48.2387215Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8dtx94il
2022-05-18T03:35:48.2387985Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8dtx94il/_remote_module_non_scriptable.py
2022-05-18T03:35:48.2395990Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpllrc_rm_
2022-05-18T03:35:48.2397682Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpllrc_rm_/_remote_module_non_scriptable.py
2022-05-18T03:35:48.2547066Z dist init r=3, world=4
2022-05-18T03:35:48.2555706Z dist init r=2, world=4
2022-05-18T03:35:48.2768692Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpznponsjd
2022-05-18T03:35:48.2769319Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpznponsjd/_remote_module_non_scriptable.py
2022-05-18T03:35:48.2926243Z dist init r=1, world=4
2022-05-18T03:35:48.2942543Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp14r3vz8v
2022-05-18T03:35:48.2944855Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp14r3vz8v/_remote_module_non_scriptable.py
2022-05-18T03:35:48.3100314Z dist init r=0, world=4
2022-05-18T03:35:48.3438189Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:35:48.3539203Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:35:48.3642707Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:48.3643406Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:35:48.3644077Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:35:48.3644808Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:48.3645333Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:48.3645854Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:48.3650760Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:35:48.3651357Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:35:48.3651890Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:35:48.3652446Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:35:48.5747998Z skip: Need at least 2 CUDA devices (0.915s)
2022-05-18T03:35:48.5790955Z   test_mixture_of_experts_offload_true_prefetch_pre_no_shard_clip_norm_type_None (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7373
2022-05-18T03:35:48.5816917Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7374
2022-05-18T03:35:48.5840104Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7375
2022-05-18T03:35:48.5864667Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7376
2022-05-18T03:35:49.1414509Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptbhgo5ku
2022-05-18T03:35:49.1415593Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptbhgo5ku/_remote_module_non_scriptable.py
2022-05-18T03:35:49.1522929Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp14zycgnv
2022-05-18T03:35:49.1524189Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp14zycgnv/_remote_module_non_scriptable.py
2022-05-18T03:35:49.1537133Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0n3fiv3v
2022-05-18T03:35:49.1538578Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0n3fiv3v/_remote_module_non_scriptable.py
2022-05-18T03:35:49.1573667Z dist init r=0, world=4
2022-05-18T03:35:49.1684224Z dist init r=3, world=4
2022-05-18T03:35:49.1700213Z dist init r=1, world=4
2022-05-18T03:35:49.2022802Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc7_cz0u8
2022-05-18T03:35:49.2024783Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc7_cz0u8/_remote_module_non_scriptable.py
2022-05-18T03:35:49.2178106Z dist init r=2, world=4
2022-05-18T03:35:49.2388279Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:35:49.2488414Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:35:49.2590841Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:35:49.2591504Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:35:49.2592373Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:49.2592903Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:49.2593431Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:49.2593951Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:49.2698008Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:35:49.2698662Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:35:49.2699272Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:35:49.2699618Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:35:49.4891503Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:35:49.4934162Z   test_mixture_of_experts_offload_true_prefetch_pre_none_clip_norm_type_2_0 (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7429
2022-05-18T03:35:49.4960403Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7430
2022-05-18T03:35:49.4982594Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7431
2022-05-18T03:35:49.5007447Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7432
2022-05-18T03:35:50.0712166Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxxg8s5jt
2022-05-18T03:35:50.0712950Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxxg8s5jt/_remote_module_non_scriptable.py
2022-05-18T03:35:50.0725651Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2bg3thg0
2022-05-18T03:35:50.0727955Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2bg3thg0/_remote_module_non_scriptable.py
2022-05-18T03:35:50.0871474Z dist init r=3, world=4
2022-05-18T03:35:50.0884899Z dist init r=1, world=4
2022-05-18T03:35:50.1158195Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpaybsrv6d
2022-05-18T03:35:50.1159425Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpaybsrv6d/_remote_module_non_scriptable.py
2022-05-18T03:35:50.1288741Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp71sze5t3
2022-05-18T03:35:50.1290698Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp71sze5t3/_remote_module_non_scriptable.py
2022-05-18T03:35:50.1313591Z dist init r=2, world=4
2022-05-18T03:35:50.1443819Z dist init r=0, world=4
2022-05-18T03:35:50.1622559Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:35:50.1786050Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:35:50.1887290Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:35:50.1887876Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:35:50.1889181Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:50.1889743Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:50.1890254Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:50.1926957Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:50.1995308Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:35:50.1995710Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:35:50.1996240Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:35:50.1996770Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:35:50.4034174Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:35:50.4077061Z   test_mixture_of_experts_offload_true_prefetch_pre_none_clip_norm_type_None (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7485
2022-05-18T03:35:50.4102796Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7486
2022-05-18T03:35:50.4126356Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7487
2022-05-18T03:35:50.4150414Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7488
2022-05-18T03:35:50.9945190Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpiu6oel5t
2022-05-18T03:35:50.9946246Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpiu6oel5t/_remote_module_non_scriptable.py
2022-05-18T03:35:51.0080304Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpojq159wz
2022-05-18T03:35:51.0082549Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpojq159wz/_remote_module_non_scriptable.py
2022-05-18T03:35:51.0104070Z dist init r=3, world=4
2022-05-18T03:35:51.0237999Z dist init r=2, world=4
2022-05-18T03:35:51.0336493Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpufe5ka81
2022-05-18T03:35:51.0337960Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpufe5ka81/_remote_module_non_scriptable.py
2022-05-18T03:35:51.0367491Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpru07b8om
2022-05-18T03:35:51.0369410Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpru07b8om/_remote_module_non_scriptable.py
2022-05-18T03:35:51.0497496Z dist init r=0, world=4
2022-05-18T03:35:51.0526388Z dist init r=1, world=4
2022-05-18T03:35:51.0714923Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:35:51.0815150Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:35:51.0917600Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:35:51.0918624Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:51.0919240Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:35:51.0919853Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:51.0920376Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:51.0921068Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:51.1024336Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:35:51.1025237Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:35:51.1025931Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:35:51.1026579Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:35:51.3176070Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:35:51.3219207Z   test_mixture_of_experts_offload_true_prefetch_pre_shard_grad_op_clip_norm_type_2_0 (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7541
2022-05-18T03:35:51.3245362Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7542
2022-05-18T03:35:51.3269196Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7543
2022-05-18T03:35:51.3294681Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7544
2022-05-18T03:35:51.9538026Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3vp6tx1p
2022-05-18T03:35:51.9539105Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3vp6tx1p/_remote_module_non_scriptable.py
2022-05-18T03:35:51.9566065Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9nuk323a
2022-05-18T03:35:51.9567552Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9nuk323a/_remote_module_non_scriptable.py
2022-05-18T03:35:51.9631703Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm11485t7
2022-05-18T03:35:51.9633564Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm11485t7/_remote_module_non_scriptable.py
2022-05-18T03:35:51.9698839Z dist init r=3, world=4
2022-05-18T03:35:51.9727511Z dist init r=0, world=4
2022-05-18T03:35:51.9789779Z dist init r=2, world=4
2022-05-18T03:35:51.9899579Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu3dj4t_t
2022-05-18T03:35:51.9901599Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu3dj4t_t/_remote_module_non_scriptable.py
2022-05-18T03:35:52.0054190Z dist init r=1, world=4
2022-05-18T03:35:52.0208927Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:35:52.0309676Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:35:52.0413027Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:35:52.0413539Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:35:52.0414409Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:52.0415125Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:52.0415920Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:52.0416749Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:52.0418737Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:35:52.0419281Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:35:52.0419791Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:35:52.0420327Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:35:52.2321760Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:35:52.2365202Z   test_mixture_of_experts_offload_true_prefetch_pre_shard_grad_op_clip_norm_type_None (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7597
2022-05-18T03:35:52.2392324Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7598
2022-05-18T03:35:52.2416857Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7599
2022-05-18T03:35:52.2440993Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7600
2022-05-18T03:35:52.8624424Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8wk55az0
2022-05-18T03:35:52.8625245Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8wk55az0/_remote_module_non_scriptable.py
2022-05-18T03:35:52.8654761Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf38w03tu
2022-05-18T03:35:52.8655449Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph5gpvyq1
2022-05-18T03:35:52.8656244Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf38w03tu/_remote_module_non_scriptable.py
2022-05-18T03:35:52.8657073Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph5gpvyq1/_remote_module_non_scriptable.py
2022-05-18T03:35:52.8744526Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyiq85nqp
2022-05-18T03:35:52.8745836Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyiq85nqp/_remote_module_non_scriptable.py
2022-05-18T03:35:52.8786551Z dist init r=2, world=4
2022-05-18T03:35:52.8817930Z dist init r=3, world=4
2022-05-18T03:35:52.8818158Z dist init r=1, world=4
2022-05-18T03:35:52.8907332Z dist init r=0, world=4
2022-05-18T03:35:52.9095266Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:35:52.9297710Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:35:52.9400457Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:35:52.9401296Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:52.9401766Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:35:52.9402251Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:52.9402832Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:52.9500400Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:52.9507882Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:35:52.9508431Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:35:52.9509041Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:35:52.9509593Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:35:53.1467607Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:35:53.1510759Z   test_mixture_of_experts_with_delay_before_free_offload_false_none_no_shard (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7653
2022-05-18T03:35:53.1537097Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7654
2022-05-18T03:35:53.1560299Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7655
2022-05-18T03:35:53.1584391Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7656
2022-05-18T03:35:53.7471414Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpewmjj9qs
2022-05-18T03:35:53.7472881Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpewmjj9qs/_remote_module_non_scriptable.py
2022-05-18T03:35:53.7548994Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpiiuqxzd2
2022-05-18T03:35:53.7551766Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpiiuqxzd2/_remote_module_non_scriptable.py
2022-05-18T03:35:53.7632298Z dist init r=2, world=4
2022-05-18T03:35:53.7706857Z dist init r=1, world=4
2022-05-18T03:35:53.7743024Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9e31kkft
2022-05-18T03:35:53.7745048Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9e31kkft/_remote_module_non_scriptable.py
2022-05-18T03:35:53.7891466Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsxfxy5we
2022-05-18T03:35:53.7893290Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsxfxy5we/_remote_module_non_scriptable.py
2022-05-18T03:35:53.7905072Z dist init r=3, world=4
2022-05-18T03:35:53.8047319Z dist init r=0, world=4
2022-05-18T03:35:53.8344699Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:35:53.8446359Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:35:53.8447051Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:35:53.8447654Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:35:53.8448773Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:53.8449686Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:53.8450279Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:53.8451068Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:53.8454879Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:35:53.8455462Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:35:53.8455845Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:35:53.8456193Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:35:54.0611093Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:35:54.0653950Z   test_mixture_of_experts_with_delay_before_free_offload_false_none_none (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7709
2022-05-18T03:35:54.0679671Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7710
2022-05-18T03:35:54.0703013Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7711
2022-05-18T03:35:54.0726684Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7712
2022-05-18T03:35:54.6685823Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9p1f2i5h
2022-05-18T03:35:54.6687034Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9p1f2i5h/_remote_module_non_scriptable.py
2022-05-18T03:35:54.6777501Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu3phbe36
2022-05-18T03:35:54.6779083Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu3phbe36/_remote_module_non_scriptable.py
2022-05-18T03:35:54.6849696Z dist init r=0, world=4
2022-05-18T03:35:54.6935735Z dist init r=1, world=4
2022-05-18T03:35:54.7137392Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkc4set8i
2022-05-18T03:35:54.7138396Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkc4set8i/_remote_module_non_scriptable.py
2022-05-18T03:35:54.7177466Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqgcwoe7q
2022-05-18T03:35:54.7179793Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqgcwoe7q/_remote_module_non_scriptable.py
2022-05-18T03:35:54.7294953Z dist init r=3, world=4
2022-05-18T03:35:54.7333652Z dist init r=2, world=4
2022-05-18T03:35:54.7543270Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:35:54.7644787Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:35:54.7645227Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:35:54.7646540Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:35:54.7647318Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:54.7648099Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:54.7648748Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:54.7649270Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:54.7752935Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:35:54.7753552Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:35:54.7753909Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:35:54.7754361Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:35:54.9753743Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:35:54.9795661Z   test_mixture_of_experts_with_delay_before_free_offload_false_none_shard_grad_op (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7765
2022-05-18T03:35:54.9822444Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7766
2022-05-18T03:35:54.9845416Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7767
2022-05-18T03:35:54.9869801Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7768
2022-05-18T03:35:55.5653422Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv6d95u6i
2022-05-18T03:35:55.5654229Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv6d95u6i/_remote_module_non_scriptable.py
2022-05-18T03:35:55.5814245Z dist init r=2, world=4
2022-05-18T03:35:55.6005879Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuctfh9h4
2022-05-18T03:35:55.6006822Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuctfh9h4/_remote_module_non_scriptable.py
2022-05-18T03:35:55.6035604Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt2bpfl62
2022-05-18T03:35:55.6037466Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt2bpfl62/_remote_module_non_scriptable.py
2022-05-18T03:35:55.6165034Z dist init r=1, world=4
2022-05-18T03:35:55.6195078Z dist init r=3, world=4
2022-05-18T03:35:55.6293878Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnhcp4ygv
2022-05-18T03:35:55.6295878Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnhcp4ygv/_remote_module_non_scriptable.py
2022-05-18T03:35:55.6449778Z dist init r=0, world=4
2022-05-18T03:35:55.6604923Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:35:55.6776815Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:35:55.6879511Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:35:55.6880365Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:55.6880786Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:35:55.6881272Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:55.6881800Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:55.6909064Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:55.6987059Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:35:55.6987888Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:35:55.6988816Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:35:55.6989316Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:35:55.8896748Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:35:55.8939549Z   test_mixture_of_experts_with_delay_before_free_offload_false_prefetch_post_no_shard (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7821
2022-05-18T03:35:55.8966032Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7822
2022-05-18T03:35:55.8989154Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7823
2022-05-18T03:35:55.9013413Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7824
2022-05-18T03:35:56.5019020Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi_1fqg9q
2022-05-18T03:35:56.5020229Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi_1fqg9q/_remote_module_non_scriptable.py
2022-05-18T03:35:56.5177123Z dist init r=1, world=4
2022-05-18T03:35:56.5241802Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv4qkxvup
2022-05-18T03:35:56.5243661Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv4qkxvup/_remote_module_non_scriptable.py
2022-05-18T03:35:56.5272847Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptgriqtb2
2022-05-18T03:35:56.5275015Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptgriqtb2/_remote_module_non_scriptable.py
2022-05-18T03:35:56.5384287Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc1go_yh0
2022-05-18T03:35:56.5386403Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc1go_yh0/_remote_module_non_scriptable.py
2022-05-18T03:35:56.5405840Z dist init r=3, world=4
2022-05-18T03:35:56.5435845Z dist init r=0, world=4
2022-05-18T03:35:56.5540364Z dist init r=2, world=4
2022-05-18T03:35:56.5749166Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:35:56.5849864Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:35:56.5850473Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:35:56.5851089Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:56.5851505Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:35:56.5854090Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:56.5854794Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:56.5855377Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:56.5957513Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:35:56.5958059Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:35:56.5958597Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:35:56.5959162Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:35:56.8040630Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:35:56.8083106Z   test_mixture_of_experts_with_delay_before_free_offload_false_prefetch_post_none (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7877
2022-05-18T03:35:56.8109173Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7878
2022-05-18T03:35:56.8132752Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7879
2022-05-18T03:35:56.8157217Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7880
2022-05-18T03:35:57.3770627Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5op02twl
2022-05-18T03:35:57.3771380Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5op02twl/_remote_module_non_scriptable.py
2022-05-18T03:35:57.3864660Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9_r8f_jm
2022-05-18T03:35:57.3866514Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9_r8f_jm/_remote_module_non_scriptable.py
2022-05-18T03:35:57.3869597Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpekh7co98
2022-05-18T03:35:57.3871919Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpekh7co98/_remote_module_non_scriptable.py
2022-05-18T03:35:57.3932137Z dist init r=3, world=4
2022-05-18T03:35:57.4022758Z dist init r=0, world=4
2022-05-18T03:35:57.4029049Z dist init r=2, world=4
2022-05-18T03:35:57.4376927Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcve9tnn_
2022-05-18T03:35:57.4378528Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcve9tnn_/_remote_module_non_scriptable.py
2022-05-18T03:35:57.4532906Z dist init r=1, world=4
2022-05-18T03:35:57.4843997Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:35:57.4844597Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:35:57.4945699Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:35:57.4946188Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:35:57.4946979Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:57.4947692Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:57.4948319Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:57.4948908Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:57.5052774Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:35:57.5053908Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:35:57.5054586Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:35:57.5055368Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:35:57.7184693Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:35:57.7238167Z   test_mixture_of_experts_with_delay_before_free_offload_false_prefetch_post_shard_grad_op (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7933
2022-05-18T03:35:57.7264339Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7934
2022-05-18T03:35:57.7287940Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7935
2022-05-18T03:35:57.7311983Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7936
2022-05-18T03:35:58.3010094Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp53mo946d
2022-05-18T03:35:58.3010870Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp53mo946d/_remote_module_non_scriptable.py
2022-05-18T03:35:58.3168512Z dist init r=1, world=4
2022-05-18T03:35:58.3447159Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbn72vph7
2022-05-18T03:35:58.3448179Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbn72vph7/_remote_module_non_scriptable.py
2022-05-18T03:35:58.3604219Z dist init r=0, world=4
2022-05-18T03:35:58.3618576Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgomgfoth
2022-05-18T03:35:58.3620006Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgomgfoth/_remote_module_non_scriptable.py
2022-05-18T03:35:58.3641867Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdtofrrzi
2022-05-18T03:35:58.3643781Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdtofrrzi/_remote_module_non_scriptable.py
2022-05-18T03:35:58.3778526Z dist init r=3, world=4
2022-05-18T03:35:58.3799367Z dist init r=2, world=4
2022-05-18T03:35:58.3987473Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:35:58.4082041Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:35:58.4082721Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:35:58.4083256Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:35:58.4083913Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:58.4084445Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:58.4084956Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:58.4089797Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:58.4191366Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:35:58.4193023Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:35:58.4194680Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:35:58.4195272Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:35:58.6338737Z skip: Need at least 2 CUDA devices (0.915s)
2022-05-18T03:35:58.6380939Z   test_mixture_of_experts_with_delay_before_free_offload_false_prefetch_pre_no_shard (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7989
2022-05-18T03:35:58.6406695Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7990
2022-05-18T03:35:58.6430465Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7991
2022-05-18T03:35:58.6453933Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7992
2022-05-18T03:35:59.2263211Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjwto9h8x
2022-05-18T03:35:59.2264934Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjwto9h8x/_remote_module_non_scriptable.py
2022-05-18T03:35:59.2387881Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphhinta5a
2022-05-18T03:35:59.2389475Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphhinta5a/_remote_module_non_scriptable.py
2022-05-18T03:35:59.2420133Z dist init r=0, world=4
2022-05-18T03:35:59.2544443Z dist init r=3, world=4
2022-05-18T03:35:59.2651180Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwe2hs0a9
2022-05-18T03:35:59.2653273Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwe2hs0a9/_remote_module_non_scriptable.py
2022-05-18T03:35:59.2697511Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpok1ulq6g
2022-05-18T03:35:59.2700055Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpok1ulq6g/_remote_module_non_scriptable.py
2022-05-18T03:35:59.2811233Z dist init r=1, world=4
2022-05-18T03:35:59.2854928Z dist init r=2, world=4
2022-05-18T03:35:59.3122490Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:35:59.3223412Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:35:59.3325848Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:35:59.3326459Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:35:59.3327371Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:59.3328052Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:59.3328931Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:59.3329720Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:35:59.3332507Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:35:59.3333430Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:35:59.3334083Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:35:59.3334483Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:35:59.5480096Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:35:59.5522457Z   test_mixture_of_experts_with_delay_before_free_offload_false_prefetch_pre_none (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8045
2022-05-18T03:35:59.5549102Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8046
2022-05-18T03:35:59.5572387Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8047
2022-05-18T03:35:59.5596345Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8048
2022-05-18T03:36:00.1194000Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv7k5od9s
2022-05-18T03:36:00.1194732Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv7k5od9s/_remote_module_non_scriptable.py
2022-05-18T03:36:00.1206480Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7eqhbppa
2022-05-18T03:36:00.1209074Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7eqhbppa/_remote_module_non_scriptable.py
2022-05-18T03:36:00.1228607Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp_h936k4
2022-05-18T03:36:00.1231481Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp_h936k4/_remote_module_non_scriptable.py
2022-05-18T03:36:00.1366238Z dist init r=3, world=4
2022-05-18T03:36:00.1437819Z dist init r=1, world=4
2022-05-18T03:36:00.1460451Z dist init r=0, world=4
2022-05-18T03:36:00.1690782Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzn5a0svj
2022-05-18T03:36:00.1692246Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzn5a0svj/_remote_module_non_scriptable.py
2022-05-18T03:36:00.1848884Z dist init r=2, world=4
2022-05-18T03:36:00.1976665Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:36:00.2149888Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:36:00.2251831Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:36:00.2252549Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:36:00.2253252Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:00.2253783Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:00.2254301Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:00.2280645Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:00.2358524Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:36:00.2359067Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:36:00.2359595Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:36:00.2360024Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:36:00.4623254Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:36:00.4666429Z   test_mixture_of_experts_with_delay_before_free_offload_false_prefetch_pre_shard_grad_op (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8101
2022-05-18T03:36:00.4692858Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8102
2022-05-18T03:36:00.4716772Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8103
2022-05-18T03:36:00.4741121Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8104
2022-05-18T03:36:01.0573630Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw7mfn105
2022-05-18T03:36:01.0574397Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw7mfn105/_remote_module_non_scriptable.py
2022-05-18T03:36:01.0659143Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpednek1zq
2022-05-18T03:36:01.0659910Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpednek1zq/_remote_module_non_scriptable.py
2022-05-18T03:36:01.0734902Z dist init r=1, world=4
2022-05-18T03:36:01.0816034Z dist init r=2, world=4
2022-05-18T03:36:01.0923095Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbmy4g4c_
2022-05-18T03:36:01.0924953Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbmy4g4c_/_remote_module_non_scriptable.py
2022-05-18T03:36:01.0927046Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbqnuhhqu
2022-05-18T03:36:01.0929308Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbqnuhhqu/_remote_module_non_scriptable.py
2022-05-18T03:36:01.1079865Z dist init r=3, world=4
2022-05-18T03:36:01.1083932Z dist init r=0, world=4
2022-05-18T03:36:01.1287685Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:36:01.1427857Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:36:01.1529873Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:36:01.1530310Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:36:01.1530968Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:01.1531489Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:01.1532011Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:01.1591906Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:01.1637111Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:36:01.1637660Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:36:01.1638166Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:36:01.1638700Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:36:01.3766223Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:36:01.3808998Z   test_mixture_of_experts_with_delay_before_free_offload_true_none_no_shard (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8157
2022-05-18T03:36:01.3834817Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8158
2022-05-18T03:36:01.3859032Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8159
2022-05-18T03:36:01.3883122Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8160
2022-05-18T03:36:01.9886220Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk1pdkxc8
2022-05-18T03:36:01.9886991Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk1pdkxc8/_remote_module_non_scriptable.py
2022-05-18T03:36:01.9893776Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8yc5khh9
2022-05-18T03:36:01.9895680Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8yc5khh9/_remote_module_non_scriptable.py
2022-05-18T03:36:01.9968118Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppdtcylhr
2022-05-18T03:36:01.9970653Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppdtcylhr/_remote_module_non_scriptable.py
2022-05-18T03:36:02.0046453Z dist init r=0, world=4
2022-05-18T03:36:02.0054776Z dist init r=2, world=4
2022-05-18T03:36:02.0124236Z dist init r=3, world=4
2022-05-18T03:36:02.0140169Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc6s1mt70
2022-05-18T03:36:02.0142368Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc6s1mt70/_remote_module_non_scriptable.py
2022-05-18T03:36:02.0295417Z dist init r=1, world=4
2022-05-18T03:36:02.0605683Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:36:02.0606097Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:36:02.0707741Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:36:02.0708402Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:36:02.0709353Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:02.0710626Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:02.0711238Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:02.0809125Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:02.0815207Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:36:02.0816099Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:36:02.0816627Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:36:02.0817182Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:36:02.2910330Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:36:02.2953704Z   test_mixture_of_experts_with_delay_before_free_offload_true_none_none (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8213
2022-05-18T03:36:02.2981711Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8214
2022-05-18T03:36:02.3006006Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8215
2022-05-18T03:36:02.3031163Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8216
2022-05-18T03:36:02.8658584Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpequvxeq6
2022-05-18T03:36:02.8659357Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpequvxeq6/_remote_module_non_scriptable.py
2022-05-18T03:36:02.8817848Z dist init r=2, world=4
2022-05-18T03:36:02.9160389Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp03qbyokf
2022-05-18T03:36:02.9161384Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp03qbyokf/_remote_module_non_scriptable.py
2022-05-18T03:36:02.9277282Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy4cuz40k
2022-05-18T03:36:02.9278248Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy4cuz40k/_remote_module_non_scriptable.py
2022-05-18T03:36:02.9328543Z dist init r=1, world=4
2022-05-18T03:36:02.9406853Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0w56npyq
2022-05-18T03:36:02.9408241Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0w56npyq/_remote_module_non_scriptable.py
2022-05-18T03:36:02.9439776Z dist init r=0, world=4
2022-05-18T03:36:02.9564051Z dist init r=3, world=4
2022-05-18T03:36:02.9852202Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:36:02.9953362Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:36:03.0054685Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:36:03.0055266Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:36:03.0056199Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:03.0057030Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:03.0057766Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:03.0058401Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:03.0162007Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:36:03.0162868Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:36:03.0163478Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:36:03.0164022Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:36:03.2057719Z skip: Need at least 2 CUDA devices (0.915s)
2022-05-18T03:36:03.2100529Z   test_mixture_of_experts_with_delay_before_free_offload_true_none_shard_grad_op (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8269
2022-05-18T03:36:03.2127076Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8270
2022-05-18T03:36:03.2150200Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8271
2022-05-18T03:36:03.2174191Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8272
2022-05-18T03:36:03.8187278Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1we1e_3c
2022-05-18T03:36:03.8188948Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1we1e_3c/_remote_module_non_scriptable.py
2022-05-18T03:36:03.8315159Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwq7g5tyw
2022-05-18T03:36:03.8316215Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwq7g5tyw/_remote_module_non_scriptable.py
2022-05-18T03:36:03.8346805Z dist init r=0, world=4
2022-05-18T03:36:03.8372838Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxtk76ruj
2022-05-18T03:36:03.8374831Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxtk76ruj/_remote_module_non_scriptable.py
2022-05-18T03:36:03.8474197Z dist init r=2, world=4
2022-05-18T03:36:03.8501483Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzt4wbk2r
2022-05-18T03:36:03.8503973Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzt4wbk2r/_remote_module_non_scriptable.py
2022-05-18T03:36:03.8534959Z dist init r=3, world=4
2022-05-18T03:36:03.8657973Z dist init r=1, world=4
2022-05-18T03:36:03.8946784Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:36:03.9046955Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:36:03.9148935Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:36:03.9149654Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:36:03.9150518Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:03.9151106Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:03.9152021Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:03.9152569Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:03.9256906Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:36:03.9257556Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:36:03.9258130Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:36:03.9258623Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:36:04.1201835Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:36:04.1245124Z   test_mixture_of_experts_with_delay_before_free_offload_true_prefetch_post_no_shard (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8325
2022-05-18T03:36:04.1271613Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8326
2022-05-18T03:36:04.1295803Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8327
2022-05-18T03:36:04.1319952Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8328
2022-05-18T03:36:04.6960288Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4969xt10
2022-05-18T03:36:04.6962014Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptmr165ro
2022-05-18T03:36:04.6962708Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4969xt10/_remote_module_non_scriptable.py
2022-05-18T03:36:04.6964695Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptmr165ro/_remote_module_non_scriptable.py
2022-05-18T03:36:04.7122394Z dist init r=1, world=4
2022-05-18T03:36:04.7122764Z dist init r=3, world=4
2022-05-18T03:36:04.7386782Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyj3q9hze
2022-05-18T03:36:04.7388025Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyj3q9hze/_remote_module_non_scriptable.py
2022-05-18T03:36:04.7543309Z dist init r=0, world=4
2022-05-18T03:36:04.7548810Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpheijwmd9
2022-05-18T03:36:04.7551125Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpheijwmd9/_remote_module_non_scriptable.py
2022-05-18T03:36:04.7703653Z dist init r=2, world=4
2022-05-18T03:36:04.7953365Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:36:04.8054875Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:36:04.8055829Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:36:04.8056615Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:04.8057294Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:36:04.8057891Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:04.8058495Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:04.8059196Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:04.8064313Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:36:04.8064885Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:36:04.8065451Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:36:04.8065938Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:36:05.0346881Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:36:05.0389361Z   test_mixture_of_experts_with_delay_before_free_offload_true_prefetch_post_none (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8381
2022-05-18T03:36:05.0415727Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8382
2022-05-18T03:36:05.0439687Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8383
2022-05-18T03:36:05.0464335Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8384
2022-05-18T03:36:05.6216120Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1u4q54tp
2022-05-18T03:36:05.6217369Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1u4q54tp/_remote_module_non_scriptable.py
2022-05-18T03:36:05.6374962Z dist init r=3, world=4
2022-05-18T03:36:05.6573293Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpiikncjsd
2022-05-18T03:36:05.6574283Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpiikncjsd/_remote_module_non_scriptable.py
2022-05-18T03:36:05.6674990Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp31qb0gkt
2022-05-18T03:36:05.6676517Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp31qb0gkt/_remote_module_non_scriptable.py
2022-05-18T03:36:05.6730107Z dist init r=2, world=4
2022-05-18T03:36:05.6767459Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcj_55171
2022-05-18T03:36:05.6769507Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcj_55171/_remote_module_non_scriptable.py
2022-05-18T03:36:05.6830632Z dist init r=1, world=4
2022-05-18T03:36:05.6923504Z dist init r=0, world=4
2022-05-18T03:36:05.7084920Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:36:05.7241480Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:36:05.7344594Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:05.7345017Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:36:05.7345443Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:36:05.7346051Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:05.7346557Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:05.7389290Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:05.7451901Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:36:05.7452579Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:36:05.7453006Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:36:05.7453343Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:36:05.9490620Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:36:05.9532975Z   test_mixture_of_experts_with_delay_before_free_offload_true_prefetch_post_shard_grad_op (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8437
2022-05-18T03:36:05.9558854Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8438
2022-05-18T03:36:05.9582389Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8439
2022-05-18T03:36:05.9606793Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8440
2022-05-18T03:36:06.5301783Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp12zrhqbu
2022-05-18T03:36:06.5304221Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp12zrhqbu/_remote_module_non_scriptable.py
2022-05-18T03:36:06.5458764Z dist init r=0, world=4
2022-05-18T03:36:06.5553790Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2svnispr
2022-05-18T03:36:06.5554745Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2svnispr/_remote_module_non_scriptable.py
2022-05-18T03:36:06.5714364Z dist init r=1, world=4
2022-05-18T03:36:06.5983579Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_kt9yovj
2022-05-18T03:36:06.5984820Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_kt9yovj/_remote_module_non_scriptable.py
2022-05-18T03:36:06.6006791Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgddi7eix
2022-05-18T03:36:06.6008592Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgddi7eix/_remote_module_non_scriptable.py
2022-05-18T03:36:06.6142766Z dist init r=2, world=4
2022-05-18T03:36:06.6165364Z dist init r=3, world=4
2022-05-18T03:36:06.6273506Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:36:06.6372689Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:36:06.6373255Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:36:06.6373911Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:36:06.6374763Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:06.6375573Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:06.6376301Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:06.6376838Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:06.6481292Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:36:06.6481973Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:36:06.6482468Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:36:06.6482967Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:36:06.8633248Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:36:06.8675745Z   test_mixture_of_experts_with_delay_before_free_offload_true_prefetch_pre_no_shard (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8493
2022-05-18T03:36:06.8701865Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8494
2022-05-18T03:36:06.8725326Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8495
2022-05-18T03:36:06.8749455Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8496
2022-05-18T03:36:07.4474643Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4f80fnki
2022-05-18T03:36:07.4475779Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4f80fnki/_remote_module_non_scriptable.py
2022-05-18T03:36:07.4636927Z dist init r=1, world=4
2022-05-18T03:36:07.4664929Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph89l8hdu
2022-05-18T03:36:07.4665861Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph89l8hdu/_remote_module_non_scriptable.py
2022-05-18T03:36:07.4826221Z dist init r=0, world=4
2022-05-18T03:36:07.5015112Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpni_umqg4
2022-05-18T03:36:07.5016045Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpni_umqg4/_remote_module_non_scriptable.py
2022-05-18T03:36:07.5064478Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc1mmdhcf
2022-05-18T03:36:07.5066349Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc1mmdhcf/_remote_module_non_scriptable.py
2022-05-18T03:36:07.5172492Z dist init r=2, world=4
2022-05-18T03:36:07.5221999Z dist init r=3, world=4
2022-05-18T03:36:07.5330422Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:36:07.5438099Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:36:07.5438793Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:36:07.5439549Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:36:07.5440226Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:07.5440759Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:07.5441264Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:07.5533863Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:07.5545906Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:36:07.5546290Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:36:07.5546629Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:36:07.5547119Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:36:07.7775985Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:36:07.7818378Z   test_mixture_of_experts_with_delay_before_free_offload_true_prefetch_pre_none (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8549
2022-05-18T03:36:07.7844242Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8550
2022-05-18T03:36:07.7867403Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8551
2022-05-18T03:36:07.7891858Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8552
2022-05-18T03:36:08.3781125Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3jb9yztv
2022-05-18T03:36:08.3782236Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3jb9yztv/_remote_module_non_scriptable.py
2022-05-18T03:36:08.3920513Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzg6wwtek
2022-05-18T03:36:08.3922412Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzg6wwtek/_remote_module_non_scriptable.py
2022-05-18T03:36:08.3940071Z dist init r=1, world=4
2022-05-18T03:36:08.3985572Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq5ggws_7
2022-05-18T03:36:08.3987529Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq5ggws_7/_remote_module_non_scriptable.py
2022-05-18T03:36:08.4080616Z dist init r=3, world=4
2022-05-18T03:36:08.4143404Z dist init r=2, world=4
2022-05-18T03:36:08.4231302Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9conw9_x
2022-05-18T03:36:08.4233168Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9conw9_x/_remote_module_non_scriptable.py
2022-05-18T03:36:08.4387160Z dist init r=0, world=4
2022-05-18T03:36:08.4590487Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:36:08.4754793Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:36:08.4856131Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:36:08.4856558Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:36:08.4857184Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:08.4857717Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:08.4858233Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:08.4894557Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:08.4963900Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:36:08.4964538Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:36:08.4965057Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:36:08.4966557Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:36:08.6918676Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:36:08.6960991Z   test_mixture_of_experts_with_delay_before_free_offload_true_prefetch_pre_shard_grad_op (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8605
2022-05-18T03:36:08.6985964Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8606
2022-05-18T03:36:08.7009164Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8607
2022-05-18T03:36:08.7033676Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8608
2022-05-18T03:36:09.2714500Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmy_r5l_n
2022-05-18T03:36:09.2715698Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmy_r5l_n/_remote_module_non_scriptable.py
2022-05-18T03:36:09.2729431Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpoo8ong90
2022-05-18T03:36:09.2731718Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpoo8ong90/_remote_module_non_scriptable.py
2022-05-18T03:36:09.2876602Z dist init r=2, world=4
2022-05-18T03:36:09.2887710Z dist init r=0, world=4
2022-05-18T03:36:09.3175967Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzwshvraq
2022-05-18T03:36:09.3176987Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzwshvraq/_remote_module_non_scriptable.py
2022-05-18T03:36:09.3324832Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqmedz6or
2022-05-18T03:36:09.3326474Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqmedz6or/_remote_module_non_scriptable.py
2022-05-18T03:36:09.3331546Z dist init r=3, world=4
2022-05-18T03:36:09.3480112Z dist init r=1, world=4
2022-05-18T03:36:09.3790554Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:36:09.3791203Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:36:09.3792217Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:09.3792772Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:36:09.3793171Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:36:09.3794074Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:09.3794602Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:09.3795122Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:09.3899537Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:36:09.3900267Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:36:09.3900663Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:36:09.3901011Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:36:09.6060563Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:36:09.6105926Z   test_nested_all_wrapped_model_offload_false_none_no_shard_clip_norm_type_2_0 (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8661
2022-05-18T03:36:09.6132613Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8662
2022-05-18T03:36:09.6156285Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8663
2022-05-18T03:36:09.6180405Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8664
2022-05-18T03:36:10.1820012Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp646ywhe2
2022-05-18T03:36:10.1820759Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgrivhk5n
2022-05-18T03:36:10.1821468Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp646ywhe2/_remote_module_non_scriptable.py
2022-05-18T03:36:10.1822216Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgrivhk5n/_remote_module_non_scriptable.py
2022-05-18T03:36:10.1977822Z dist init r=2, world=4
2022-05-18T03:36:10.1978169Z dist init r=3, world=4
2022-05-18T03:36:10.2240276Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi2bzkn9o
2022-05-18T03:36:10.2240998Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi2bzkn9o/_remote_module_non_scriptable.py
2022-05-18T03:36:10.2327171Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw00jchzg
2022-05-18T03:36:10.2328998Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw00jchzg/_remote_module_non_scriptable.py
2022-05-18T03:36:10.2400700Z dist init r=0, world=4
2022-05-18T03:36:10.2485894Z dist init r=1, world=4
2022-05-18T03:36:10.2790285Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:36:10.2811847Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:36:10.2914222Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:36:10.2915296Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:10.2915717Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:36:10.2916205Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:10.2916725Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:10.2993469Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:10.3022420Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:36:10.3023203Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:36:10.3023727Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:36:10.3024282Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:36:10.5206906Z skip: Need at least 2 CUDA devices (0.915s)
2022-05-18T03:36:10.5250363Z   test_nested_all_wrapped_model_offload_false_none_no_shard_clip_norm_type_None (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8717
2022-05-18T03:36:10.5275956Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8718
2022-05-18T03:36:10.5299958Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8719
2022-05-18T03:36:10.5324606Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8720
2022-05-18T03:36:11.1024527Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3vuokekg
2022-05-18T03:36:11.1025534Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3vuokekg/_remote_module_non_scriptable.py
2022-05-18T03:36:11.1184153Z dist init r=0, world=4
2022-05-18T03:36:11.1313406Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd7nl_uq5
2022-05-18T03:36:11.1314346Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd7nl_uq5/_remote_module_non_scriptable.py
2022-05-18T03:36:11.1314773Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptxe9zfu1
2022-05-18T03:36:11.1318139Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptxe9zfu1/_remote_module_non_scriptable.py
2022-05-18T03:36:11.1472001Z dist init r=1, world=4
2022-05-18T03:36:11.1472233Z dist init r=3, world=4
2022-05-18T03:36:11.1773898Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplodu4jef
2022-05-18T03:36:11.1775543Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplodu4jef/_remote_module_non_scriptable.py
2022-05-18T03:36:11.1927805Z dist init r=2, world=4
2022-05-18T03:36:11.2084909Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:36:11.2185793Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:36:11.2286416Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:36:11.2287306Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:36:11.2288083Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:11.2288704Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:11.2289391Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:11.2290137Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:11.2394650Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:36:11.2395132Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:36:11.2395640Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:36:11.2396171Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:36:11.4351077Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:36:11.4393078Z   test_nested_all_wrapped_model_offload_false_none_none_clip_norm_type_2_0 (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8773
2022-05-18T03:36:11.4418559Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8774
2022-05-18T03:36:11.4441985Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8775
2022-05-18T03:36:11.4465802Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8776
2022-05-18T03:36:12.0509888Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7knzk03r
2022-05-18T03:36:12.0510623Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7knzk03r/_remote_module_non_scriptable.py
2022-05-18T03:36:12.0574198Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl2nfiu9x
2022-05-18T03:36:12.0576042Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl2nfiu9x/_remote_module_non_scriptable.py
2022-05-18T03:36:12.0668408Z dist init r=1, world=4
2022-05-18T03:36:12.0704297Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplmg360k8
2022-05-18T03:36:12.0706095Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplmg360k8/_remote_module_non_scriptable.py
2022-05-18T03:36:12.0732401Z dist init r=2, world=4
2022-05-18T03:36:12.0792186Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwi7im4b9
2022-05-18T03:36:12.0794504Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwi7im4b9/_remote_module_non_scriptable.py
2022-05-18T03:36:12.0861172Z dist init r=0, world=4
2022-05-18T03:36:12.0949467Z dist init r=3, world=4
2022-05-18T03:36:12.1144000Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:36:12.1244924Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:36:12.1346588Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:36:12.1347555Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:12.1348173Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:36:12.1348956Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:12.1349753Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:12.1350280Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:12.1354204Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:36:12.1354726Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:36:12.1355205Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:36:12.1357087Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:36:12.3492688Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:36:12.3536552Z   test_nested_all_wrapped_model_offload_false_none_none_clip_norm_type_None (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8829
2022-05-18T03:36:12.3562948Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8830
2022-05-18T03:36:12.3586358Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8831
2022-05-18T03:36:12.3610416Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8832
2022-05-18T03:36:12.9258417Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpflo64hf4
2022-05-18T03:36:12.9259098Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpflo64hf4/_remote_module_non_scriptable.py
2022-05-18T03:36:12.9364857Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgs9z9a_2
2022-05-18T03:36:12.9365815Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgs9z9a_2/_remote_module_non_scriptable.py
2022-05-18T03:36:12.9420208Z dist init r=1, world=4
2022-05-18T03:36:12.9525332Z dist init r=3, world=4
2022-05-18T03:36:12.9718019Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkb41oeso
2022-05-18T03:36:12.9718745Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkb41oeso/_remote_module_non_scriptable.py
2022-05-18T03:36:12.9846638Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxz_slfms
2022-05-18T03:36:12.9847579Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxz_slfms/_remote_module_non_scriptable.py
2022-05-18T03:36:12.9877183Z dist init r=2, world=4
2022-05-18T03:36:13.0003095Z dist init r=0, world=4
2022-05-18T03:36:13.0186783Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:36:13.0389714Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:36:13.0491966Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:36:13.0493131Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:13.0493759Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:36:13.0494485Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:13.0495013Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:13.0495534Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:13.0499506Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:36:13.0500788Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:36:13.0502338Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:36:13.0503350Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:36:13.2638195Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:36:13.2681463Z   test_nested_all_wrapped_model_offload_false_none_shard_grad_op_clip_norm_type_2_0 (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8885
2022-05-18T03:36:13.2707833Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8886
2022-05-18T03:36:13.2731556Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8887
2022-05-18T03:36:13.2756104Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8888
2022-05-18T03:36:13.8323101Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2h780ixo
2022-05-18T03:36:13.8323904Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2h780ixo/_remote_module_non_scriptable.py
2022-05-18T03:36:13.8480814Z dist init r=2, world=4
2022-05-18T03:36:13.8850027Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptz2sny95
2022-05-18T03:36:13.8850780Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptz2sny95/_remote_module_non_scriptable.py
2022-05-18T03:36:13.8854198Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgq4oqqho
2022-05-18T03:36:13.8856384Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgq4oqqho/_remote_module_non_scriptable.py
2022-05-18T03:36:13.9005146Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0fbeu36j
2022-05-18T03:36:13.9007481Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0fbeu36j/_remote_module_non_scriptable.py
2022-05-18T03:36:13.9009142Z dist init r=1, world=4
2022-05-18T03:36:13.9009428Z dist init r=3, world=4
2022-05-18T03:36:13.9162188Z dist init r=0, world=4
2022-05-18T03:36:13.9318984Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:36:13.9495519Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:36:13.9596635Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:36:13.9599885Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:36:13.9600600Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:13.9622783Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:13.9699489Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:13.9700421Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:13.9707434Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:36:13.9708124Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:36:13.9708700Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:36:13.9709219Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:36:14.1782735Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:36:14.1825482Z   test_nested_all_wrapped_model_offload_false_none_shard_grad_op_clip_norm_type_None (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8941
2022-05-18T03:36:14.1851620Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8942
2022-05-18T03:36:14.1875563Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8943
2022-05-18T03:36:14.1899694Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8944
2022-05-18T03:36:14.7746358Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcc99s91p
2022-05-18T03:36:14.7747144Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcc99s91p/_remote_module_non_scriptable.py
2022-05-18T03:36:14.7874229Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg1ec2983
2022-05-18T03:36:14.7876569Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg1ec2983/_remote_module_non_scriptable.py
2022-05-18T03:36:14.7908753Z dist init r=3, world=4
2022-05-18T03:36:14.8032629Z dist init r=1, world=4
2022-05-18T03:36:14.8079324Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgg9oxir5
2022-05-18T03:36:14.8081189Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgg9oxir5/_remote_module_non_scriptable.py
2022-05-18T03:36:14.8235939Z dist init r=2, world=4
2022-05-18T03:36:14.8328978Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3v867m2a
2022-05-18T03:36:14.8330870Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3v867m2a/_remote_module_non_scriptable.py
2022-05-18T03:36:14.8484518Z dist init r=0, world=4
2022-05-18T03:36:14.8722323Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:36:14.8823771Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:36:14.8926772Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:36:14.8928018Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:14.8928819Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:36:14.8929740Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:14.8930344Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:14.8930866Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:14.8935259Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:36:14.8935925Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:36:14.8936466Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:36:14.8937230Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:36:15.0926567Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:36:15.0970096Z   test_nested_all_wrapped_model_offload_false_prefetch_post_no_shard_clip_norm_type_2_0 (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8997
2022-05-18T03:36:15.0996017Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8998
2022-05-18T03:36:15.1019185Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8999
2022-05-18T03:36:15.1043544Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9000
2022-05-18T03:36:15.7168326Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4e48jq5v
2022-05-18T03:36:15.7169099Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmmhtwxzq
2022-05-18T03:36:15.7171401Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4e48jq5v/_remote_module_non_scriptable.py
2022-05-18T03:36:15.7172456Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmmhtwxzq/_remote_module_non_scriptable.py
2022-05-18T03:36:15.7325654Z dist init r=1, world=4
2022-05-18T03:36:15.7330357Z dist init r=2, world=4
2022-05-18T03:36:15.7347134Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpurq187y9
2022-05-18T03:36:15.7349271Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpurq187y9/_remote_module_non_scriptable.py
2022-05-18T03:36:15.7429291Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpoyiy4l86
2022-05-18T03:36:15.7430765Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpoyiy4l86/_remote_module_non_scriptable.py
2022-05-18T03:36:15.7506774Z dist init r=0, world=4
2022-05-18T03:36:15.7585312Z dist init r=3, world=4
2022-05-18T03:36:15.7919613Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:36:15.7920367Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:36:15.7921006Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:36:15.7922097Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:15.7922832Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:36:15.7923402Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:15.7923908Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:15.7924428Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:15.8027183Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:36:15.8027641Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:36:15.8028147Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:36:15.8028719Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:36:16.0070151Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:36:16.0114813Z   test_nested_all_wrapped_model_offload_false_prefetch_post_no_shard_clip_norm_type_None (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9053
2022-05-18T03:36:16.0140819Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9054
2022-05-18T03:36:16.0164368Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9055
2022-05-18T03:36:16.0188424Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9056
2022-05-18T03:36:16.5860576Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprbzr0k09
2022-05-18T03:36:16.5861979Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprbzr0k09/_remote_module_non_scriptable.py
2022-05-18T03:36:16.6019163Z dist init r=0, world=4
2022-05-18T03:36:16.6083584Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyk3istgz
2022-05-18T03:36:16.6084816Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyk3istgz/_remote_module_non_scriptable.py
2022-05-18T03:36:16.6242567Z dist init r=1, world=4
2022-05-18T03:36:16.6385785Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvt82jjmh
2022-05-18T03:36:16.6387283Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvt82jjmh/_remote_module_non_scriptable.py
2022-05-18T03:36:16.6518025Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl6fca_c_
2022-05-18T03:36:16.6519099Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl6fca_c_/_remote_module_non_scriptable.py
2022-05-18T03:36:16.6544295Z dist init r=2, world=4
2022-05-18T03:36:16.6674964Z dist init r=3, world=4
2022-05-18T03:36:16.6783455Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:36:16.6854332Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:36:16.6955509Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:36:16.6956199Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:36:16.6957213Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:16.6957739Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:16.6958276Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:16.6986994Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:16.7064478Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:36:16.7065500Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:36:16.7066070Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:36:16.7066585Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:36:16.9215370Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:36:16.9258516Z   test_nested_all_wrapped_model_offload_false_prefetch_post_none_clip_norm_type_2_0 (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9109
2022-05-18T03:36:16.9284611Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9110
2022-05-18T03:36:16.9308333Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9111
2022-05-18T03:36:16.9332639Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9112
2022-05-18T03:36:17.4948299Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp17ml6_f2
2022-05-18T03:36:17.4949319Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp17ml6_f2/_remote_module_non_scriptable.py
2022-05-18T03:36:17.5111004Z dist init r=3, world=4
2022-05-18T03:36:17.5501871Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy7q46mcw
2022-05-18T03:36:17.5502428Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy7q46mcw/_remote_module_non_scriptable.py
2022-05-18T03:36:17.5519361Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpangdj5sa
2022-05-18T03:36:17.5521871Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpangdj5sa/_remote_module_non_scriptable.py
2022-05-18T03:36:17.5595052Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp780a9saw
2022-05-18T03:36:17.5597291Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp780a9saw/_remote_module_non_scriptable.py
2022-05-18T03:36:17.5660434Z dist init r=2, world=4
2022-05-18T03:36:17.5677954Z dist init r=0, world=4
2022-05-18T03:36:17.5753565Z dist init r=1, world=4
2022-05-18T03:36:17.5970830Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:36:17.6071089Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:36:17.6173214Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:36:17.6174187Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:17.6174820Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:36:17.6175466Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:17.6175991Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:17.6176516Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:17.6281136Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:36:17.6281702Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:36:17.6282272Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:36:17.6282782Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:36:17.8359196Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:36:17.8401943Z   test_nested_all_wrapped_model_offload_false_prefetch_post_none_clip_norm_type_None (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9165
2022-05-18T03:36:17.8427773Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9166
2022-05-18T03:36:17.8451197Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9167
2022-05-18T03:36:17.8475631Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9168
2022-05-18T03:36:18.4251039Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzs5_2aqj
2022-05-18T03:36:18.4252270Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzs5_2aqj/_remote_module_non_scriptable.py
2022-05-18T03:36:18.4414854Z dist init r=1, world=4
2022-05-18T03:36:18.4603817Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf9asb8lq
2022-05-18T03:36:18.4604735Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf9asb8lq/_remote_module_non_scriptable.py
2022-05-18T03:36:18.4692752Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv1mwrxx8
2022-05-18T03:36:18.4693291Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc7xq78y3
2022-05-18T03:36:18.4694425Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv1mwrxx8/_remote_module_non_scriptable.py
2022-05-18T03:36:18.4695128Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc7xq78y3/_remote_module_non_scriptable.py
2022-05-18T03:36:18.4761799Z dist init r=0, world=4
2022-05-18T03:36:18.4852854Z dist init r=3, world=4
2022-05-18T03:36:18.4853153Z dist init r=2, world=4
2022-05-18T03:36:18.5062701Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:36:18.5164144Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:36:18.5164813Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:36:18.5165519Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:36:18.5166497Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:18.5167437Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:18.5168197Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:18.5168776Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:18.5272016Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:36:18.5272725Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:36:18.5273364Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:36:18.5273714Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:36:18.7503245Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:36:18.7545750Z   test_nested_all_wrapped_model_offload_false_prefetch_post_shard_grad_op_clip_norm_type_2_0 (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9221
2022-05-18T03:36:18.7572270Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9222
2022-05-18T03:36:18.7596528Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9223
2022-05-18T03:36:18.7620992Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9224
2022-05-18T03:36:19.3473375Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpksj6ux88
2022-05-18T03:36:19.3474396Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpksj6ux88/_remote_module_non_scriptable.py
2022-05-18T03:36:19.3572992Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj5w4zv_u
2022-05-18T03:36:19.3574438Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj5w4zv_u/_remote_module_non_scriptable.py
2022-05-18T03:36:19.3631548Z dist init r=3, world=4
2022-05-18T03:36:19.3732549Z dist init r=0, world=4
2022-05-18T03:36:19.3776793Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz6ig2_9j
2022-05-18T03:36:19.3778641Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz6ig2_9j/_remote_module_non_scriptable.py
2022-05-18T03:36:19.3887474Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo3t77e83
2022-05-18T03:36:19.3889506Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo3t77e83/_remote_module_non_scriptable.py
2022-05-18T03:36:19.3933336Z dist init r=1, world=4
2022-05-18T03:36:19.4043993Z dist init r=2, world=4
2022-05-18T03:36:19.4253351Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:36:19.4344287Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:36:19.4446625Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:36:19.4447481Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:19.4447945Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:36:19.4448905Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:19.4449492Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:19.4456581Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:19.4554957Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:36:19.4555661Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:36:19.4556127Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:36:19.4556533Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:36:19.6646621Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:36:19.6690057Z   test_nested_all_wrapped_model_offload_false_prefetch_post_shard_grad_op_clip_norm_type_None (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9277
2022-05-18T03:36:19.6715721Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9278
2022-05-18T03:36:19.6738428Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9279
2022-05-18T03:36:19.6762627Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9280
2022-05-18T03:36:20.2521291Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp575elfel
2022-05-18T03:36:20.2522035Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp575elfel/_remote_module_non_scriptable.py
2022-05-18T03:36:20.2667288Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqk8igaoe
2022-05-18T03:36:20.2668304Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqk8igaoe/_remote_module_non_scriptable.py
2022-05-18T03:36:20.2679566Z dist init r=3, world=4
2022-05-18T03:36:20.2824702Z dist init r=1, world=4
2022-05-18T03:36:20.2916699Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm2qpz2te
2022-05-18T03:36:20.2918414Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm2qpz2te/_remote_module_non_scriptable.py
2022-05-18T03:36:20.3005152Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpiwby59li
2022-05-18T03:36:20.3007111Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpiwby59li/_remote_module_non_scriptable.py
2022-05-18T03:36:20.3072442Z dist init r=2, world=4
2022-05-18T03:36:20.3163750Z dist init r=0, world=4
2022-05-18T03:36:20.3382260Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:36:20.3483323Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:36:20.3586213Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:36:20.3586834Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:36:20.3587766Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:20.3588794Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:20.3589315Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:20.3589840Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:20.3692199Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:36:20.3692938Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:36:20.3693702Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:36:20.3694360Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:36:20.5788744Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:36:20.5833212Z   test_nested_all_wrapped_model_offload_false_prefetch_pre_no_shard_clip_norm_type_2_0 (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9333
2022-05-18T03:36:20.5858913Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9334
2022-05-18T03:36:20.5881669Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9335
2022-05-18T03:36:20.5905864Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9336
2022-05-18T03:36:21.1891808Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp20yuiwfp
2022-05-18T03:36:21.1892794Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp20yuiwfp/_remote_module_non_scriptable.py
2022-05-18T03:36:21.1975336Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsdovz25y
2022-05-18T03:36:21.1976517Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsdovz25y/_remote_module_non_scriptable.py
2022-05-18T03:36:21.2054539Z dist init r=0, world=4
2022-05-18T03:36:21.2132818Z dist init r=2, world=4
2022-05-18T03:36:21.2174480Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_d9t8pb5
2022-05-18T03:36:21.2176841Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_d9t8pb5/_remote_module_non_scriptable.py
2022-05-18T03:36:21.2329745Z dist init r=1, world=4
2022-05-18T03:36:21.2354359Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyinbwutc
2022-05-18T03:36:21.2356160Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyinbwutc/_remote_module_non_scriptable.py
2022-05-18T03:36:21.2510899Z dist init r=3, world=4
2022-05-18T03:36:21.2618548Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:36:21.2667924Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:36:21.2668622Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:36:21.2669298Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:36:21.2670366Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:21.2671069Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:21.2671577Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:21.2720915Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:21.2775715Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:36:21.2776310Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:36:21.2776857Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:36:21.2777360Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:36:21.4932773Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:36:21.4974654Z   test_nested_all_wrapped_model_offload_false_prefetch_pre_no_shard_clip_norm_type_None (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9389
2022-05-18T03:36:21.5000849Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9390
2022-05-18T03:36:21.5024481Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9391
2022-05-18T03:36:21.5049103Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9392
2022-05-18T03:36:22.0843852Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9fu1wcfa
2022-05-18T03:36:22.0845023Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9fu1wcfa/_remote_module_non_scriptable.py
2022-05-18T03:36:22.1005112Z dist init r=3, world=4
2022-05-18T03:36:22.1024687Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0gqtwsao
2022-05-18T03:36:22.1026487Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0gqtwsao/_remote_module_non_scriptable.py
2022-05-18T03:36:22.1184085Z dist init r=2, world=4
2022-05-18T03:36:22.1293179Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuvp07xpk
2022-05-18T03:36:22.1295196Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuvp07xpk/_remote_module_non_scriptable.py
2022-05-18T03:36:22.1447735Z dist init r=0, world=4
2022-05-18T03:36:22.1499249Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgz8phkzh
2022-05-18T03:36:22.1501424Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgz8phkzh/_remote_module_non_scriptable.py
2022-05-18T03:36:22.1653677Z dist init r=1, world=4
2022-05-18T03:36:22.1918757Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:36:22.2019889Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:36:22.2020564Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:36:22.2021481Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:22.2022093Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:36:22.2022651Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:22.2023418Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:22.2023942Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:22.2126521Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:36:22.2127153Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:36:22.2127703Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:36:22.2128399Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:36:22.4076036Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:36:22.4119005Z   test_nested_all_wrapped_model_offload_false_prefetch_pre_none_clip_norm_type_2_0 (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9445
2022-05-18T03:36:22.4146594Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9446
2022-05-18T03:36:22.4171109Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9447
2022-05-18T03:36:22.4196153Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9448
2022-05-18T03:36:22.9876984Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc930ry8t
2022-05-18T03:36:22.9877740Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc930ry8t/_remote_module_non_scriptable.py
2022-05-18T03:36:23.0038708Z dist init r=2, world=4
2022-05-18T03:36:23.0142748Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk1abq_nr
2022-05-18T03:36:23.0144148Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk1abq_nr/_remote_module_non_scriptable.py
2022-05-18T03:36:23.0244765Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpna31dya_
2022-05-18T03:36:23.0246696Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpna31dya_/_remote_module_non_scriptable.py
2022-05-18T03:36:23.0300352Z dist init r=3, world=4
2022-05-18T03:36:23.0336045Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw8pdtss2
2022-05-18T03:36:23.0337999Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw8pdtss2/_remote_module_non_scriptable.py
2022-05-18T03:36:23.0403143Z dist init r=0, world=4
2022-05-18T03:36:23.0493171Z dist init r=1, world=4
2022-05-18T03:36:23.0804078Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:36:23.0905149Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:36:23.1007348Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:36:23.1007947Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:36:23.1008802Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:23.1009322Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:23.1009845Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:23.1010367Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:23.1114411Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:36:23.1114991Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:36:23.1115530Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:36:23.1116065Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:36:23.3223162Z skip: Need at least 2 CUDA devices (0.915s)
2022-05-18T03:36:23.3265937Z   test_nested_all_wrapped_model_offload_false_prefetch_pre_none_clip_norm_type_None (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9501
2022-05-18T03:36:23.3292697Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9502
2022-05-18T03:36:23.3315933Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9503
2022-05-18T03:36:23.3340458Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9504
2022-05-18T03:36:23.8956520Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjjo1wwoz
2022-05-18T03:36:23.8959366Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjjo1wwoz/_remote_module_non_scriptable.py
2022-05-18T03:36:23.9113822Z dist init r=1, world=4
2022-05-18T03:36:23.9456376Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpow8c2wpv
2022-05-18T03:36:23.9457063Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5x3di234
2022-05-18T03:36:23.9458206Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpow8c2wpv/_remote_module_non_scriptable.py
2022-05-18T03:36:23.9458897Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5x3di234/_remote_module_non_scriptable.py
2022-05-18T03:36:23.9489554Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz2bxjyfr
2022-05-18T03:36:23.9490744Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz2bxjyfr/_remote_module_non_scriptable.py
2022-05-18T03:36:23.9615217Z dist init r=0, world=4
2022-05-18T03:36:23.9615546Z dist init r=3, world=4
2022-05-18T03:36:23.9645366Z dist init r=2, world=4
2022-05-18T03:36:23.9824448Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:36:23.9925290Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:36:24.0027512Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:36:24.0028526Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:36:24.0029430Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:24.0029984Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:24.0030517Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:24.0031019Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:24.0134822Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:36:24.0135525Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:36:24.0136040Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:36:24.0136599Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:36:24.2367929Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:36:24.2411720Z   test_nested_all_wrapped_model_offload_false_prefetch_pre_shard_grad_op_clip_norm_type_2_0 (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9557
2022-05-18T03:36:24.2437557Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9558
2022-05-18T03:36:24.2461027Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9559
2022-05-18T03:36:24.2485462Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9560
2022-05-18T03:36:24.8666506Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfi6p00m5
2022-05-18T03:36:24.8667494Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfi6p00m5/_remote_module_non_scriptable.py
2022-05-18T03:36:24.8674392Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptv78kzqm
2022-05-18T03:36:24.8675988Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppsfis2e7
2022-05-18T03:36:24.8676695Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptv78kzqm/_remote_module_non_scriptable.py
2022-05-18T03:36:24.8679231Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppsfis2e7/_remote_module_non_scriptable.py
2022-05-18T03:36:24.8708142Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp32xahwyr
2022-05-18T03:36:24.8710135Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp32xahwyr/_remote_module_non_scriptable.py
2022-05-18T03:36:24.8830614Z dist init r=2, world=4
2022-05-18T03:36:24.8836114Z dist init r=3, world=4
2022-05-18T03:36:24.8837975Z dist init r=0, world=4
2022-05-18T03:36:24.8868219Z dist init r=1, world=4
2022-05-18T03:36:24.9148388Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:36:24.9249478Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:36:24.9250056Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:36:24.9250453Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:36:24.9251360Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:24.9251892Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:24.9252412Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:24.9252909Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:24.9357128Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:36:24.9357883Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:36:24.9358428Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:36:24.9359128Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:36:25.1513168Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:36:25.1555387Z   test_nested_all_wrapped_model_offload_false_prefetch_pre_shard_grad_op_clip_norm_type_None (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9613
2022-05-18T03:36:25.1581305Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9614
2022-05-18T03:36:25.1605450Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9615
2022-05-18T03:36:25.1629277Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9616
2022-05-18T03:36:25.7521995Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv2bwfk6y
2022-05-18T03:36:25.7524188Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv2bwfk6y/_remote_module_non_scriptable.py
2022-05-18T03:36:25.7609795Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6hacd12x
2022-05-18T03:36:25.7612174Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6hacd12x/_remote_module_non_scriptable.py
2022-05-18T03:36:25.7676780Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxpiyfs6a
2022-05-18T03:36:25.7678592Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxpiyfs6a/_remote_module_non_scriptable.py
2022-05-18T03:36:25.7679265Z dist init r=3, world=4
2022-05-18T03:36:25.7771533Z dist init r=0, world=4
2022-05-18T03:36:25.7835459Z dist init r=1, world=4
2022-05-18T03:36:25.8034604Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp03kcivug
2022-05-18T03:36:25.8036144Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp03kcivug/_remote_module_non_scriptable.py
2022-05-18T03:36:25.8188476Z dist init r=2, world=4
2022-05-18T03:36:25.8391660Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:36:25.8483956Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:36:25.8485222Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:25.8485687Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:36:25.8486100Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:36:25.8486624Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:25.8487143Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:25.8494017Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:25.8592734Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:36:25.8593471Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:36:25.8593840Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:36:25.8594279Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:36:26.0654934Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:36:26.0698829Z   test_nested_all_wrapped_model_offload_true_none_no_shard_clip_norm_type_2_0 (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9669
2022-05-18T03:36:26.0724616Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9670
2022-05-18T03:36:26.0748535Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9671
2022-05-18T03:36:26.0772691Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9672
2022-05-18T03:36:26.6431079Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp62rddt50
2022-05-18T03:36:26.6431835Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp62rddt50/_remote_module_non_scriptable.py
2022-05-18T03:36:26.6438590Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzcyl8m5r
2022-05-18T03:36:26.6440571Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzcyl8m5r/_remote_module_non_scriptable.py
2022-05-18T03:36:26.6588982Z dist init r=3, world=4
2022-05-18T03:36:26.6596342Z dist init r=1, world=4
2022-05-18T03:36:26.7069668Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyxammh48
2022-05-18T03:36:26.7070747Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyxammh48/_remote_module_non_scriptable.py
2022-05-18T03:36:26.7116745Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqzsebo1g
2022-05-18T03:36:26.7118971Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqzsebo1g/_remote_module_non_scriptable.py
2022-05-18T03:36:26.7229515Z dist init r=0, world=4
2022-05-18T03:36:26.7273193Z dist init r=2, world=4
2022-05-18T03:36:26.7407965Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:36:26.7501753Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:36:26.7583742Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:36:26.7584222Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:36:26.7584957Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:26.7585494Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:26.7604202Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:26.7611325Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:26.7710550Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:36:26.7711103Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:36:26.7711660Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:36:26.7712173Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:36:26.9799226Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:36:26.9841972Z   test_nested_all_wrapped_model_offload_true_none_no_shard_clip_norm_type_None (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9725
2022-05-18T03:36:26.9867739Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9726
2022-05-18T03:36:26.9890828Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9727
2022-05-18T03:36:26.9914929Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9728
2022-05-18T03:36:27.5679095Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp870ycfgf
2022-05-18T03:36:27.5679886Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp870ycfgf/_remote_module_non_scriptable.py
2022-05-18T03:36:27.5838056Z dist init r=2, world=4
2022-05-18T03:36:27.5918051Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzcr5atge
2022-05-18T03:36:27.5920147Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzcr5atge/_remote_module_non_scriptable.py
2022-05-18T03:36:27.6075365Z dist init r=1, world=4
2022-05-18T03:36:27.6106016Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_44qyloq
2022-05-18T03:36:27.6108065Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_44qyloq/_remote_module_non_scriptable.py
2022-05-18T03:36:27.6261754Z dist init r=0, world=4
2022-05-18T03:36:27.6324548Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu9updye4
2022-05-18T03:36:27.6326616Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu9updye4/_remote_module_non_scriptable.py
2022-05-18T03:36:27.6479716Z dist init r=3, world=4
2022-05-18T03:36:27.6672796Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:36:27.6673468Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:36:27.6674592Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:27.6675171Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:36:27.6675699Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:36:27.6676345Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:27.6677011Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:27.6677681Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:27.6682671Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:36:27.6683445Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:36:27.6684241Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:36:27.6684888Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:36:27.8941439Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:36:27.8984111Z   test_nested_all_wrapped_model_offload_true_none_none_clip_norm_type_2_0 (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9781
2022-05-18T03:36:27.9009924Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9782
2022-05-18T03:36:27.9033577Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9783
2022-05-18T03:36:27.9057085Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9784
2022-05-18T03:36:28.4855986Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr4446ey7
2022-05-18T03:36:28.4856712Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7odmcn_i
2022-05-18T03:36:28.4857672Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr4446ey7/_remote_module_non_scriptable.py
2022-05-18T03:36:28.4860916Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7odmcn_i/_remote_module_non_scriptable.py
2022-05-18T03:36:28.5015787Z dist init r=2, world=4
2022-05-18T03:36:28.5016129Z dist init r=3, world=4
2022-05-18T03:36:28.5257537Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqay0ihja
2022-05-18T03:36:28.5258629Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4ekur1ai
2022-05-18T03:36:28.5259313Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqay0ihja/_remote_module_non_scriptable.py
2022-05-18T03:36:28.5260732Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4ekur1ai/_remote_module_non_scriptable.py
2022-05-18T03:36:28.5414529Z dist init r=0, world=4
2022-05-18T03:36:28.5416880Z dist init r=1, world=4
2022-05-18T03:36:28.5824477Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:36:28.5925582Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:36:28.6028187Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:36:28.6029270Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:28.6029928Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:36:28.6030831Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:28.6031545Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:28.6032310Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:28.6036201Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:36:28.6036777Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:36:28.6037295Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:36:28.6037819Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:36:28.8084716Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:36:28.8127981Z   test_nested_all_wrapped_model_offload_true_none_none_clip_norm_type_None (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9837
2022-05-18T03:36:28.8153741Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9838
2022-05-18T03:36:28.8176783Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9839
2022-05-18T03:36:28.8200794Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9840
2022-05-18T03:36:29.3874746Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp209te1r7
2022-05-18T03:36:29.3875476Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp209te1r7/_remote_module_non_scriptable.py
2022-05-18T03:36:29.3876093Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4b0jqz55
2022-05-18T03:36:29.3877072Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4b0jqz55/_remote_module_non_scriptable.py
2022-05-18T03:36:29.4031621Z dist init r=1, world=4
2022-05-18T03:36:29.4033869Z dist init r=2, world=4
2022-05-18T03:36:29.4083711Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1_871ciw
2022-05-18T03:36:29.4085824Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1_871ciw/_remote_module_non_scriptable.py
2022-05-18T03:36:29.4243207Z dist init r=0, world=4
2022-05-18T03:36:29.4479845Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2s4de0t6
2022-05-18T03:36:29.4481660Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2s4de0t6/_remote_module_non_scriptable.py
2022-05-18T03:36:29.4637865Z dist init r=3, world=4
2022-05-18T03:36:29.4845277Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:36:29.4845943Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:36:29.4846427Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:36:29.4847448Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:29.4847867Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:36:29.4848356Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:29.4848882Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:29.4849401Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:29.4953301Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:36:29.4954263Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:36:29.4955112Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:36:29.4955650Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:36:29.7227883Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:36:29.7270686Z   test_nested_all_wrapped_model_offload_true_none_shard_grad_op_clip_norm_type_2_0 (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9893
2022-05-18T03:36:29.7297002Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9894
2022-05-18T03:36:29.7320333Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9895
2022-05-18T03:36:29.7344597Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9896
2022-05-18T03:36:30.2998378Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfwuc63s8
2022-05-18T03:36:30.2999489Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuqr6td4p
2022-05-18T03:36:30.3000473Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfwuc63s8/_remote_module_non_scriptable.py
2022-05-18T03:36:30.3001429Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuqr6td4p/_remote_module_non_scriptable.py
2022-05-18T03:36:30.3156971Z dist init r=2, world=4
2022-05-18T03:36:30.3157319Z dist init r=0, world=4
2022-05-18T03:36:30.3367033Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdsnzpnrr
2022-05-18T03:36:30.3367993Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdsnzpnrr/_remote_module_non_scriptable.py
2022-05-18T03:36:30.3519973Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm13b2max
2022-05-18T03:36:30.3521328Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm13b2max/_remote_module_non_scriptable.py
2022-05-18T03:36:30.3521852Z dist init r=1, world=4
2022-05-18T03:36:30.3679078Z dist init r=3, world=4
2022-05-18T03:36:30.3869287Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:36:30.3870185Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:36:30.3871108Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:36:30.3872693Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:30.3873371Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:30.3873848Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:36:30.3874342Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:30.3874864Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:30.3976857Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:36:30.3977549Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:36:30.3977914Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:36:30.3978270Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:36:30.6370798Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:36:30.6413222Z   test_nested_all_wrapped_model_offload_true_none_shard_grad_op_clip_norm_type_None (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9949
2022-05-18T03:36:30.6440093Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9950
2022-05-18T03:36:30.6463933Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9951
2022-05-18T03:36:30.6494002Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9952
2022-05-18T03:36:31.2397771Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuiy5yd88
2022-05-18T03:36:31.2398615Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuiy5yd88/_remote_module_non_scriptable.py
2022-05-18T03:36:31.2474192Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwcj23h4d
2022-05-18T03:36:31.2475250Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwcj23h4d/_remote_module_non_scriptable.py
2022-05-18T03:36:31.2558009Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptd6hvc5o
2022-05-18T03:36:31.2558568Z dist init r=1, world=4
2022-05-18T03:36:31.2559050Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptd6hvc5o/_remote_module_non_scriptable.py
2022-05-18T03:36:31.2634289Z dist init r=3, world=4
2022-05-18T03:36:31.2717980Z dist init r=0, world=4
2022-05-18T03:36:31.2813701Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkwu_djix
2022-05-18T03:36:31.2815679Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkwu_djix/_remote_module_non_scriptable.py
2022-05-18T03:36:31.2969541Z dist init r=2, world=4
2022-05-18T03:36:31.3178515Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:36:31.3272388Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:36:31.3373596Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:36:31.3374204Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:36:31.3374836Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:31.3375857Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:31.3376440Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:31.3381542Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:31.3481897Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:36:31.3482430Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:36:31.3482964Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:36:31.3483516Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:36:31.5520210Z skip: Need at least 2 CUDA devices (0.915s)
2022-05-18T03:36:31.5564263Z   test_nested_all_wrapped_model_offload_true_prefetch_post_no_shard_clip_norm_type_2_0 (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10005
2022-05-18T03:36:31.5590690Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10006
2022-05-18T03:36:31.5613952Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10007
2022-05-18T03:36:31.5639030Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10008
2022-05-18T03:36:32.1534227Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxmqir3tf
2022-05-18T03:36:32.1534986Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxmqir3tf/_remote_module_non_scriptable.py
2022-05-18T03:36:32.1603750Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4enh0xal
2022-05-18T03:36:32.1605603Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4enh0xal/_remote_module_non_scriptable.py
2022-05-18T03:36:32.1697558Z dist init r=0, world=4
2022-05-18T03:36:32.1761368Z dist init r=3, world=4
2022-05-18T03:36:32.1820611Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm94xy8gf
2022-05-18T03:36:32.1822738Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm94xy8gf/_remote_module_non_scriptable.py
2022-05-18T03:36:32.1889747Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplffplpmx
2022-05-18T03:36:32.1891789Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplffplpmx/_remote_module_non_scriptable.py
2022-05-18T03:36:32.1981800Z dist init r=1, world=4
2022-05-18T03:36:32.2045378Z dist init r=2, world=4
2022-05-18T03:36:32.2293070Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:36:32.2293697Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:36:32.2294196Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:36:32.2295073Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:32.2295639Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:36:32.2296260Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:32.2296895Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:32.2297451Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:32.2399770Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:36:32.2400446Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:36:32.2400970Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:36:32.2401508Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:36:32.4665424Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:36:32.4708848Z   test_nested_all_wrapped_model_offload_true_prefetch_post_no_shard_clip_norm_type_None (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10061
2022-05-18T03:36:32.4735046Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10062
2022-05-18T03:36:32.4758217Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10063
2022-05-18T03:36:32.4782549Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10064
2022-05-18T03:36:33.0391610Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyaq82zdo
2022-05-18T03:36:33.0392357Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyaq82zdo/_remote_module_non_scriptable.py
2022-05-18T03:36:33.0549962Z dist init r=2, world=4
2022-05-18T03:36:33.0565072Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppi_ckyw1
2022-05-18T03:36:33.0566410Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppi_ckyw1/_remote_module_non_scriptable.py
2022-05-18T03:36:33.0722310Z dist init r=1, world=4
2022-05-18T03:36:33.0962916Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu1kut_pg
2022-05-18T03:36:33.0963799Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu1kut_pg/_remote_module_non_scriptable.py
2022-05-18T03:36:33.1013261Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp692qrzz5
2022-05-18T03:36:33.1014854Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp692qrzz5/_remote_module_non_scriptable.py
2022-05-18T03:36:33.1117772Z dist init r=0, world=4
2022-05-18T03:36:33.1168922Z dist init r=3, world=4
2022-05-18T03:36:33.1362184Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:36:33.1464520Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:36:33.1465119Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:36:33.1465648Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:36:33.1466280Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:33.1466793Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:33.1467313Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:33.1467838Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:33.1574688Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:36:33.1575112Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:36:33.1575507Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:36:33.1576006Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:36:33.3808095Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:36:33.3850933Z   test_nested_all_wrapped_model_offload_true_prefetch_post_none_clip_norm_type_2_0 (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10117
2022-05-18T03:36:33.3876680Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10118
2022-05-18T03:36:33.3899766Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10119
2022-05-18T03:36:33.3924382Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10120
2022-05-18T03:36:33.9742313Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4b64hwxa
2022-05-18T03:36:33.9743433Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4b64hwxa/_remote_module_non_scriptable.py
2022-05-18T03:36:33.9790178Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkfva_iif
2022-05-18T03:36:33.9791880Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkfva_iif/_remote_module_non_scriptable.py
2022-05-18T03:36:33.9830395Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps4_z99mj
2022-05-18T03:36:33.9832023Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps4_z99mj/_remote_module_non_scriptable.py
2022-05-18T03:36:33.9901733Z dist init r=1, world=4
2022-05-18T03:36:33.9949789Z dist init r=3, world=4
2022-05-18T03:36:33.9986634Z dist init r=2, world=4
2022-05-18T03:36:34.0135105Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7bgb5d9x
2022-05-18T03:36:34.0136677Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7bgb5d9x/_remote_module_non_scriptable.py
2022-05-18T03:36:34.0289635Z dist init r=0, world=4
2022-05-18T03:36:34.0598784Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:36:34.0700170Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:36:34.0802565Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:36:34.0803944Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:34.0805293Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:36:34.0806332Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:34.0806911Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:34.0807708Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:34.0810867Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:36:34.0811462Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:36:34.0812013Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:36:34.0812543Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:36:34.2952138Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:36:34.2995025Z   test_nested_all_wrapped_model_offload_true_prefetch_post_none_clip_norm_type_None (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10173
2022-05-18T03:36:34.3021783Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10174
2022-05-18T03:36:34.3045403Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10175
2022-05-18T03:36:34.3069881Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10176
2022-05-18T03:36:34.8936877Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp583fa1iv
2022-05-18T03:36:34.8937643Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp583fa1iv/_remote_module_non_scriptable.py
2022-05-18T03:36:34.9006816Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeae64rid
2022-05-18T03:36:34.9007889Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeae64rid/_remote_module_non_scriptable.py
2022-05-18T03:36:34.9097814Z dist init r=2, world=4
2022-05-18T03:36:34.9164589Z dist init r=3, world=4
2022-05-18T03:36:34.9181832Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphqyu4r4i
2022-05-18T03:36:34.9184227Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphqyu4r4i/_remote_module_non_scriptable.py
2022-05-18T03:36:34.9340943Z dist init r=1, world=4
2022-05-18T03:36:34.9410759Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphmiq9ztz
2022-05-18T03:36:34.9413432Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphmiq9ztz/_remote_module_non_scriptable.py
2022-05-18T03:36:34.9568056Z dist init r=0, world=4
2022-05-18T03:36:34.9953989Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:36:35.0056188Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:36:35.0057217Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:35.0057820Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:36:35.0058267Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:36:35.0058767Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:35.0059293Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:35.0059815Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:35.0068151Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:36:35.0068588Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:36:35.0068938Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:36:35.0069295Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:36:35.2096373Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:36:35.2139044Z   test_nested_all_wrapped_model_offload_true_prefetch_post_shard_grad_op_clip_norm_type_2_0 (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10229
2022-05-18T03:36:35.2165285Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10230
2022-05-18T03:36:35.2188970Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10231
2022-05-18T03:36:35.2213450Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10232
2022-05-18T03:36:35.8060373Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmzx48sdu
2022-05-18T03:36:35.8061162Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmzx48sdu/_remote_module_non_scriptable.py
2022-05-18T03:36:35.8217289Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptzngsfeo
2022-05-18T03:36:35.8218044Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptzngsfeo/_remote_module_non_scriptable.py
2022-05-18T03:36:35.8218517Z dist init r=1, world=4
2022-05-18T03:36:35.8378759Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7toaj5cc
2022-05-18T03:36:35.8379497Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkcnhnzt6
2022-05-18T03:36:35.8380114Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkcnhnzt6/_remote_module_non_scriptable.py
2022-05-18T03:36:35.8380528Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7toaj5cc/_remote_module_non_scriptable.py
2022-05-18T03:36:35.8382753Z dist init r=0, world=4
2022-05-18T03:36:35.8537699Z dist init r=2, world=4
2022-05-18T03:36:35.8542688Z dist init r=3, world=4
2022-05-18T03:36:35.8651612Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:36:35.8752294Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:36:35.8853515Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:36:35.8854187Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:36:35.8854844Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:35.8855524Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:35.8856037Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:35.8856559Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:35.8961792Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:36:35.8962493Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:36:35.8962983Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:36:35.8963513Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:36:36.1241076Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:36:36.1285229Z   test_nested_all_wrapped_model_offload_true_prefetch_post_shard_grad_op_clip_norm_type_None (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10285
2022-05-18T03:36:36.1311568Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10286
2022-05-18T03:36:36.1335329Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10287
2022-05-18T03:36:36.1358730Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10288
2022-05-18T03:36:36.7633438Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpupk27wuz
2022-05-18T03:36:36.7634634Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpupk27wuz/_remote_module_non_scriptable.py
2022-05-18T03:36:36.7736073Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1w34zc28
2022-05-18T03:36:36.7737254Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1w34zc28/_remote_module_non_scriptable.py
2022-05-18T03:36:36.7778707Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp44cyvk_h
2022-05-18T03:36:36.7780470Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp44cyvk_h/_remote_module_non_scriptable.py
2022-05-18T03:36:36.7795029Z dist init r=2, world=4
2022-05-18T03:36:36.7882317Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1zox9n5r
2022-05-18T03:36:36.7884363Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1zox9n5r/_remote_module_non_scriptable.py
2022-05-18T03:36:36.7898103Z dist init r=1, world=4
2022-05-18T03:36:36.7937789Z dist init r=3, world=4
2022-05-18T03:36:36.8039611Z dist init r=0, world=4
2022-05-18T03:36:36.8348716Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:36:36.8551491Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:36:36.8653197Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:36:36.8653907Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:36:36.8654627Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:36.8655661Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:36.8656249Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:36.8754243Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:36.8765422Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:36:36.8766004Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:36:36.8766633Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:36:36.8767290Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:36:37.0385185Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:36:37.0428248Z   test_nested_all_wrapped_model_offload_true_prefetch_pre_no_shard_clip_norm_type_2_0 (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10341
2022-05-18T03:36:37.0454618Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10342
2022-05-18T03:36:37.0478737Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10343
2022-05-18T03:36:37.0503565Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10344
2022-05-18T03:36:37.6226441Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp60je56n4
2022-05-18T03:36:37.6227657Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp60je56n4/_remote_module_non_scriptable.py
2022-05-18T03:36:37.6333696Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3rsqr0nx
2022-05-18T03:36:37.6334915Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3rsqr0nx/_remote_module_non_scriptable.py
2022-05-18T03:36:37.6390405Z dist init r=2, world=4
2022-05-18T03:36:37.6492245Z dist init r=3, world=4
2022-05-18T03:36:37.6654080Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpik4z8nk_
2022-05-18T03:36:37.6655424Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpik4z8nk_/_remote_module_non_scriptable.py
2022-05-18T03:36:37.6815984Z dist init r=0, world=4
2022-05-18T03:36:37.6918015Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpegijk_vu
2022-05-18T03:36:37.6919560Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpegijk_vu/_remote_module_non_scriptable.py
2022-05-18T03:36:37.7077714Z dist init r=1, world=4
2022-05-18T03:36:37.7204320Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:36:37.7388327Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:36:37.7388889Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:36:37.7389317Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:36:37.7389955Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:37.7390530Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:37.7391102Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:37.7407430Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:37.7495713Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:36:37.7496188Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:36:37.7496992Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:36:37.7497615Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:36:37.9530055Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:36:37.9585614Z   test_nested_all_wrapped_model_offload_true_prefetch_pre_no_shard_clip_norm_type_None (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10397
2022-05-18T03:36:37.9612658Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10398
2022-05-18T03:36:37.9637138Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10399
2022-05-18T03:36:37.9662242Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10400
2022-05-18T03:36:38.5253416Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpste8ox3t
2022-05-18T03:36:38.5254360Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpste8ox3t/_remote_module_non_scriptable.py
2022-05-18T03:36:38.5313585Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3i_8hhdb
2022-05-18T03:36:38.5315142Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3i_8hhdb/_remote_module_non_scriptable.py
2022-05-18T03:36:38.5367775Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa7apmblp
2022-05-18T03:36:38.5369204Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa7apmblp/_remote_module_non_scriptable.py
2022-05-18T03:36:38.5418524Z dist init r=2, world=4
2022-05-18T03:36:38.5473268Z dist init r=1, world=4
2022-05-18T03:36:38.5526971Z dist init r=0, world=4
2022-05-18T03:36:38.5802439Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjkijxpyz
2022-05-18T03:36:38.5804664Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjkijxpyz/_remote_module_non_scriptable.py
2022-05-18T03:36:38.5956809Z dist init r=3, world=4
2022-05-18T03:36:38.6086087Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:36:38.6086733Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:36:38.6087178Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:36:38.6087609Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:36:38.6088469Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:38.6089117Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:38.6089707Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:38.6090719Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:38.6194517Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:36:38.6195112Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:36:38.6195646Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:36:38.6196204Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:36:38.7686471Z skip: Need at least 2 CUDA devices (0.816s)
2022-05-18T03:36:38.7728735Z   test_nested_all_wrapped_model_offload_true_prefetch_pre_none_clip_norm_type_2_0 (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10453
2022-05-18T03:36:38.7754734Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10454
2022-05-18T03:36:38.7777840Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10455
2022-05-18T03:36:38.7802014Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10456
2022-05-18T03:36:39.3762929Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj3yk_qc5
2022-05-18T03:36:39.3763684Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj3yk_qc5/_remote_module_non_scriptable.py
2022-05-18T03:36:39.3790109Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjtt74n1t
2022-05-18T03:36:39.3791590Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjtt74n1t/_remote_module_non_scriptable.py
2022-05-18T03:36:39.3921457Z dist init r=1, world=4
2022-05-18T03:36:39.3948856Z dist init r=3, world=4
2022-05-18T03:36:39.4076733Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphtq9tctz
2022-05-18T03:36:39.4078361Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphtq9tctz/_remote_module_non_scriptable.py
2022-05-18T03:36:39.4093837Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8jjqh_iv
2022-05-18T03:36:39.4096053Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8jjqh_iv/_remote_module_non_scriptable.py
2022-05-18T03:36:39.4235515Z dist init r=0, world=4
2022-05-18T03:36:39.4253657Z dist init r=2, world=4
2022-05-18T03:36:39.4458720Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:36:39.4564043Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:36:39.4666091Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:36:39.4666887Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:39.4667312Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:36:39.4667814Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:39.4668417Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:39.4762492Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:39.4773622Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:36:39.4774171Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:36:39.4774712Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:36:39.4775270Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:36:39.6828575Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:36:39.6872569Z   test_nested_all_wrapped_model_offload_true_prefetch_pre_none_clip_norm_type_None (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10509
2022-05-18T03:36:39.6898731Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10510
2022-05-18T03:36:39.6923527Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10511
2022-05-18T03:36:39.6947944Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10512
2022-05-18T03:36:40.2588556Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkhnwpcek
2022-05-18T03:36:40.2589598Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkhnwpcek/_remote_module_non_scriptable.py
2022-05-18T03:36:40.2748487Z dist init r=2, world=4
2022-05-18T03:36:40.3146717Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8wo5q6nc
2022-05-18T03:36:40.3147697Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8wo5q6nc/_remote_module_non_scriptable.py
2022-05-18T03:36:40.3304125Z dist init r=0, world=4
2022-05-18T03:36:40.3409779Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnspet79u
2022-05-18T03:36:40.3410624Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnspet79u/_remote_module_non_scriptable.py
2022-05-18T03:36:40.3567843Z dist init r=3, world=4
2022-05-18T03:36:40.3705702Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw9atf5w7
2022-05-18T03:36:40.3707415Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw9atf5w7/_remote_module_non_scriptable.py
2022-05-18T03:36:40.3859563Z dist init r=1, world=4
2022-05-18T03:36:40.4218674Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:36:40.4320355Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:36:40.4320953Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:36:40.4321475Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:36:40.4322128Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:40.4322646Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:40.4323327Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:40.4323848Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:40.4429467Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:36:40.4430086Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:36:40.4430659Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:36:40.4431177Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:36:40.5974698Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:36:40.6029867Z   test_nested_all_wrapped_model_offload_true_prefetch_pre_shard_grad_op_clip_norm_type_2_0 (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10565
2022-05-18T03:36:40.6055692Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10566
2022-05-18T03:36:40.6078902Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10567
2022-05-18T03:36:40.6103421Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10568
2022-05-18T03:36:41.1881540Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppftfhlga
2022-05-18T03:36:41.1882474Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppftfhlga/_remote_module_non_scriptable.py
2022-05-18T03:36:41.2042450Z dist init r=3, world=4
2022-05-18T03:36:41.2081095Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt017ar22
2022-05-18T03:36:41.2082892Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt017ar22/_remote_module_non_scriptable.py
2022-05-18T03:36:41.2237836Z dist init r=1, world=4
2022-05-18T03:36:41.2310570Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5ueh354s
2022-05-18T03:36:41.2312696Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5ueh354s/_remote_module_non_scriptable.py
2022-05-18T03:36:41.2313400Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxz4xg55e
2022-05-18T03:36:41.2315442Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxz4xg55e/_remote_module_non_scriptable.py
2022-05-18T03:36:41.2466871Z dist init r=2, world=4
2022-05-18T03:36:41.2471137Z dist init r=0, world=4
2022-05-18T03:36:41.2777348Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:36:41.2850120Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:36:41.2951587Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:36:41.2952125Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:36:41.2959428Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:41.2960062Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:41.2960595Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:41.2979766Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:41.3059412Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:36:41.3059960Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:36:41.3060506Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:36:41.3061056Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:36:41.5130430Z skip: Need at least 2 CUDA devices (0.915s)
2022-05-18T03:36:41.5174812Z   test_nested_all_wrapped_model_offload_true_prefetch_pre_shard_grad_op_clip_norm_type_None (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10621
2022-05-18T03:36:41.5201712Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10622
2022-05-18T03:36:41.5226255Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10623
2022-05-18T03:36:41.5250471Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10624
2022-05-18T03:36:42.1206362Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmbl4k4dl
2022-05-18T03:36:42.1207176Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmbl4k4dl/_remote_module_non_scriptable.py
2022-05-18T03:36:42.1349636Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppvutyd7v
2022-05-18T03:36:42.1352598Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppvutyd7v/_remote_module_non_scriptable.py
2022-05-18T03:36:42.1362222Z dist init r=0, world=4
2022-05-18T03:36:42.1429905Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmkx0xr8z
2022-05-18T03:36:42.1431747Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmkx0xr8z/_remote_module_non_scriptable.py
2022-05-18T03:36:42.1508730Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd59pibre
2022-05-18T03:36:42.1510820Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd59pibre/_remote_module_non_scriptable.py
2022-05-18T03:36:42.1512561Z dist init r=1, world=4
2022-05-18T03:36:42.1589511Z dist init r=3, world=4
2022-05-18T03:36:42.1665754Z dist init r=2, world=4
2022-05-18T03:36:42.1798479Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:36:42.1975194Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:36:42.1975768Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:36:42.1976248Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:36:42.1977199Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:42.1977732Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:42.1978260Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:42.2002005Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:42.2082935Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:36:42.2083333Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:36:42.2083760Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:36:42.2085144Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:36:42.4277724Z skip: Need at least 2 CUDA devices (0.915s)
2022-05-18T03:36:42.4319860Z   test_nested_wrapped_model_offload_false_none_no_shard (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10677
2022-05-18T03:36:42.4346223Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10678
2022-05-18T03:36:42.4369735Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10679
2022-05-18T03:36:42.4394296Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10680
2022-05-18T03:36:43.0368167Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8znt1qfn
2022-05-18T03:36:43.0369209Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8znt1qfn/_remote_module_non_scriptable.py
2022-05-18T03:36:43.0529933Z dist init r=2, world=4
2022-05-18T03:36:43.0607914Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe4ujw_qc
2022-05-18T03:36:43.0610538Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe4ujw_qc/_remote_module_non_scriptable.py
2022-05-18T03:36:43.0765833Z dist init r=1, world=4
2022-05-18T03:36:43.1525378Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmponp9rnmc
2022-05-18T03:36:43.1525992Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmponp9rnmc/_remote_module_non_scriptable.py
2022-05-18T03:36:43.1612056Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjwa4tb79
2022-05-18T03:36:43.1614083Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjwa4tb79/_remote_module_non_scriptable.py
2022-05-18T03:36:43.1680565Z dist init r=3, world=4
2022-05-18T03:36:43.1771473Z dist init r=0, world=4
2022-05-18T03:36:43.2083087Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:36:43.2083514Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:36:43.2184245Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:36:43.2184955Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:36:43.2185574Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:43.2186107Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:43.2188352Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:43.2188886Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:43.2291373Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:36:43.2292094Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:36:43.2292588Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:36:43.2292993Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:36:43.4422125Z skip: Need at least 2 CUDA devices (1.014s)
2022-05-18T03:36:43.4465781Z   test_nested_wrapped_model_offload_false_none_none (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10733
2022-05-18T03:36:43.4491695Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10734
2022-05-18T03:36:43.4515878Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10735
2022-05-18T03:36:43.4540473Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10736
2022-05-18T03:36:44.0399382Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqostqakj
2022-05-18T03:36:44.0400386Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqostqakj/_remote_module_non_scriptable.py
2022-05-18T03:36:44.0558029Z dist init r=0, world=4
2022-05-18T03:36:44.0597199Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzm18wrga
2022-05-18T03:36:44.0598195Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzm18wrga/_remote_module_non_scriptable.py
2022-05-18T03:36:44.0656742Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr4rexqww
2022-05-18T03:36:44.0658742Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr4rexqww/_remote_module_non_scriptable.py
2022-05-18T03:36:44.0668408Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqerhk_lv
2022-05-18T03:36:44.0670344Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqerhk_lv/_remote_module_non_scriptable.py
2022-05-18T03:36:44.0757015Z dist init r=1, world=4
2022-05-18T03:36:44.0815224Z dist init r=2, world=4
2022-05-18T03:36:44.0826098Z dist init r=3, world=4
2022-05-18T03:36:44.0933538Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:36:44.1034622Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:36:44.1135944Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:36:44.1136523Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:36:44.1137357Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:44.1138177Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:44.1138783Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:44.1139313Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:44.1242893Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:36:44.1243460Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:36:44.1244014Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:36:44.1244560Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:36:44.3567530Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:36:44.3610009Z   test_nested_wrapped_model_offload_false_none_shard_grad_op (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10789
2022-05-18T03:36:44.3636439Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10790
2022-05-18T03:36:44.3660891Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10791
2022-05-18T03:36:44.3684883Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10792
2022-05-18T03:36:44.9638748Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt24a7l0k
2022-05-18T03:36:44.9639493Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt24a7l0k/_remote_module_non_scriptable.py
2022-05-18T03:36:44.9716719Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2j7q2l77
2022-05-18T03:36:44.9718084Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2j7q2l77/_remote_module_non_scriptable.py
2022-05-18T03:36:44.9720707Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi6hu8h4k
2022-05-18T03:36:44.9723596Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi6hu8h4k/_remote_module_non_scriptable.py
2022-05-18T03:36:44.9805026Z dist init r=3, world=4
2022-05-18T03:36:44.9875894Z dist init r=0, world=4
2022-05-18T03:36:44.9880420Z dist init r=1, world=4
2022-05-18T03:36:45.0071527Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeprql2xg
2022-05-18T03:36:45.0073392Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeprql2xg/_remote_module_non_scriptable.py
2022-05-18T03:36:45.0231315Z dist init r=2, world=4
2022-05-18T03:36:45.0488548Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:36:45.0488977Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:36:45.0589878Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:36:45.0590770Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:36:45.0591751Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:45.0592556Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:45.0593133Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:45.0593633Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:45.0697013Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:36:45.0697532Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:36:45.0698076Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:36:45.0698601Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:36:45.2711834Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:36:45.2753884Z   test_nested_wrapped_model_offload_false_prefetch_post_no_shard (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10845
2022-05-18T03:36:45.2780070Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10846
2022-05-18T03:36:45.2804002Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10847
2022-05-18T03:36:45.2828428Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10848
2022-05-18T03:36:45.8767088Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmm3s55v_
2022-05-18T03:36:45.8768048Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmm3s55v_/_remote_module_non_scriptable.py
2022-05-18T03:36:45.8926983Z dist init r=0, world=4
2022-05-18T03:36:45.9029524Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpevhdz26y
2022-05-18T03:36:45.9030919Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpevhdz26y/_remote_module_non_scriptable.py
2022-05-18T03:36:45.9185551Z dist init r=1, world=4
2022-05-18T03:36:45.9321396Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2txsqs7m
2022-05-18T03:36:45.9322988Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2txsqs7m/_remote_module_non_scriptable.py
2022-05-18T03:36:45.9476898Z dist init r=3, world=4
2022-05-18T03:36:45.9683172Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpknj9dgdp
2022-05-18T03:36:45.9684673Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpknj9dgdp/_remote_module_non_scriptable.py
2022-05-18T03:36:45.9837813Z dist init r=2, world=4
2022-05-18T03:36:46.0047507Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:36:46.0141369Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:36:46.0142623Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:36:46.0143579Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:36:46.0144269Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:46.0144867Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:46.0149932Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:46.0243450Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:46.0250894Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:36:46.0251875Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:36:46.0252419Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:36:46.0253795Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:36:46.1854615Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:36:46.1896356Z   test_nested_wrapped_model_offload_false_prefetch_post_none (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10901
2022-05-18T03:36:46.1921797Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10902
2022-05-18T03:36:46.1946239Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10903
2022-05-18T03:36:46.1970489Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10904
2022-05-18T03:36:46.7716269Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4k85ez2b
2022-05-18T03:36:46.7717049Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4k85ez2b/_remote_module_non_scriptable.py
2022-05-18T03:36:46.7837300Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsc9_ky2u
2022-05-18T03:36:46.7838543Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsc9_ky2u/_remote_module_non_scriptable.py
2022-05-18T03:36:46.7879485Z dist init r=2, world=4
2022-05-18T03:36:46.7996891Z dist init r=3, world=4
2022-05-18T03:36:46.8101782Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc433dvru
2022-05-18T03:36:46.8103968Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc433dvru/_remote_module_non_scriptable.py
2022-05-18T03:36:46.8262737Z dist init r=1, world=4
2022-05-18T03:36:46.8356257Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9ebfk23p
2022-05-18T03:36:46.8358871Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9ebfk23p/_remote_module_non_scriptable.py
2022-05-18T03:36:46.8512069Z dist init r=0, world=4
2022-05-18T03:36:46.8810059Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:36:46.8910994Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:36:46.9013472Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:36:46.9014661Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:46.9015286Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:36:46.9016071Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:46.9016860Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:46.9017691Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:46.9020361Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:36:46.9020855Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:36:46.9021377Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:36:46.9021908Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:36:47.0996832Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:36:47.1038601Z   test_nested_wrapped_model_offload_false_prefetch_post_shard_grad_op (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10957
2022-05-18T03:36:47.1065112Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10958
2022-05-18T03:36:47.1088722Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10959
2022-05-18T03:36:47.1113148Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10960
2022-05-18T03:36:47.6757481Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwitlcim2
2022-05-18T03:36:47.6758274Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwitlcim2/_remote_module_non_scriptable.py
2022-05-18T03:36:47.6919300Z dist init r=3, world=4
2022-05-18T03:36:47.6969612Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp06eppz6x
2022-05-18T03:36:47.6971227Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp06eppz6x/_remote_module_non_scriptable.py
2022-05-18T03:36:47.7127560Z dist init r=2, world=4
2022-05-18T03:36:47.7307452Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7svzrec3
2022-05-18T03:36:47.7308549Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7svzrec3/_remote_module_non_scriptable.py
2022-05-18T03:36:47.7378709Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl1a9567c
2022-05-18T03:36:47.7380917Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl1a9567c/_remote_module_non_scriptable.py
2022-05-18T03:36:47.7466224Z dist init r=1, world=4
2022-05-18T03:36:47.7535806Z dist init r=0, world=4
2022-05-18T03:36:47.7738856Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:36:47.7877666Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:36:47.7878307Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:36:47.7879508Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:47.7880097Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:36:47.7880955Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:47.7881488Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:47.7942061Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:47.7987691Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:36:47.7988446Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:36:47.7989106Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:36:47.7989757Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:36:48.0140723Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:36:48.0183576Z   test_nested_wrapped_model_offload_false_prefetch_pre_no_shard (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11013
2022-05-18T03:36:48.0212489Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11014
2022-05-18T03:36:48.0239533Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11015
2022-05-18T03:36:48.0265865Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11016
2022-05-18T03:36:48.6083592Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgmt97lxg
2022-05-18T03:36:48.6084349Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgmt97lxg/_remote_module_non_scriptable.py
2022-05-18T03:36:48.6199578Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphl9_zqmz
2022-05-18T03:36:48.6200948Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphl9_zqmz/_remote_module_non_scriptable.py
2022-05-18T03:36:48.6241212Z dist init r=0, world=4
2022-05-18T03:36:48.6357614Z dist init r=3, world=4
2022-05-18T03:36:48.6394484Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo2z3i019
2022-05-18T03:36:48.6396659Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo2z3i019/_remote_module_non_scriptable.py
2022-05-18T03:36:48.6553376Z dist init r=1, world=4
2022-05-18T03:36:48.6720636Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbbwy47l5
2022-05-18T03:36:48.6722611Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbbwy47l5/_remote_module_non_scriptable.py
2022-05-18T03:36:48.6874348Z dist init r=2, world=4
2022-05-18T03:36:48.7166621Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:36:48.7167378Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:36:48.7167980Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:36:48.7168871Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:48.7169496Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:36:48.7170120Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:48.7170857Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:48.7171645Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:48.7175706Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:36:48.7176276Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:36:48.7176821Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:36:48.7177394Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:36:48.9291886Z skip: Need at least 2 CUDA devices (0.915s)
2022-05-18T03:36:48.9333292Z   test_nested_wrapped_model_offload_false_prefetch_pre_none (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11069
2022-05-18T03:36:48.9359462Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11070
2022-05-18T03:36:48.9383591Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11071
2022-05-18T03:36:48.9408168Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11072
2022-05-18T03:36:49.5263977Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp04ea5vvi
2022-05-18T03:36:49.5265006Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp04ea5vvi/_remote_module_non_scriptable.py
2022-05-18T03:36:49.5322988Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu8j5go_a
2022-05-18T03:36:49.5324469Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu8j5go_a/_remote_module_non_scriptable.py
2022-05-18T03:36:49.5425539Z dist init r=1, world=4
2022-05-18T03:36:49.5483745Z dist init r=2, world=4
2022-05-18T03:36:49.5550504Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdbwywj5t
2022-05-18T03:36:49.5552045Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdbwywj5t/_remote_module_non_scriptable.py
2022-05-18T03:36:49.5706316Z dist init r=0, world=4
2022-05-18T03:36:49.5779170Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7_0q430_
2022-05-18T03:36:49.5781017Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7_0q430_/_remote_module_non_scriptable.py
2022-05-18T03:36:49.5935578Z dist init r=3, world=4
2022-05-18T03:36:49.6043237Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:36:49.6138543Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:36:49.6240615Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:36:49.6241220Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:36:49.6241882Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:49.6242467Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:49.6243001Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:49.6246631Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:49.6348928Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:36:49.6349769Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:36:49.6350649Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:36:49.6351145Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:36:49.8434836Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:36:49.8477716Z   test_nested_wrapped_model_offload_false_prefetch_pre_shard_grad_op (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11125
2022-05-18T03:36:49.8504494Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11126
2022-05-18T03:36:49.8527825Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11127
2022-05-18T03:36:49.8552568Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11128
2022-05-18T03:36:50.4472858Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxvgf9mag
2022-05-18T03:36:50.4473817Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxvgf9mag/_remote_module_non_scriptable.py
2022-05-18T03:36:50.4632669Z dist init r=1, world=4
2022-05-18T03:36:50.4737345Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphopde60w
2022-05-18T03:36:50.4738851Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphopde60w/_remote_module_non_scriptable.py
2022-05-18T03:36:50.4781399Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg7z1xzpp
2022-05-18T03:36:50.4783163Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg7z1xzpp/_remote_module_non_scriptable.py
2022-05-18T03:36:50.4855133Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjlribp56
2022-05-18T03:36:50.4857082Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjlribp56/_remote_module_non_scriptable.py
2022-05-18T03:36:50.4895748Z dist init r=0, world=4
2022-05-18T03:36:50.4938731Z dist init r=2, world=4
2022-05-18T03:36:50.5015129Z dist init r=3, world=4
2022-05-18T03:36:50.5122700Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:36:50.5249194Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:36:50.5249805Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:36:50.5250299Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:36:50.5250937Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:50.5251452Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:50.5252991Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:50.5326349Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:50.5359284Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:36:50.5359826Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:36:50.5360404Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:36:50.5360966Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:36:50.7579071Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:36:50.7620978Z   test_nested_wrapped_model_offload_true_none_no_shard (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11181
2022-05-18T03:36:50.7647978Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11182
2022-05-18T03:36:50.7671717Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11183
2022-05-18T03:36:50.7695934Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11184
2022-05-18T03:36:51.3595030Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp_5nkksj
2022-05-18T03:36:51.3596067Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp_5nkksj/_remote_module_non_scriptable.py
2022-05-18T03:36:51.3666772Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4pssoitm
2022-05-18T03:36:51.3668038Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4pssoitm/_remote_module_non_scriptable.py
2022-05-18T03:36:51.3755650Z dist init r=3, world=4
2022-05-18T03:36:51.3824562Z dist init r=2, world=4
2022-05-18T03:36:51.3962722Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8a0n6kpp
2022-05-18T03:36:51.3964556Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8a0n6kpp/_remote_module_non_scriptable.py
2022-05-18T03:36:51.4004026Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps88rzd_h
2022-05-18T03:36:51.4006297Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps88rzd_h/_remote_module_non_scriptable.py
2022-05-18T03:36:51.4121690Z dist init r=0, world=4
2022-05-18T03:36:51.4160706Z dist init r=1, world=4
2022-05-18T03:36:51.4367628Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:36:51.4537352Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:36:51.4640230Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:36:51.4640778Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:36:51.4641421Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:51.4641956Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:51.4642469Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:51.4670781Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:51.4748777Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:36:51.4749381Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:36:51.4749909Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:36:51.4750442Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:36:51.6722466Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:36:51.6765337Z   test_nested_wrapped_model_offload_true_none_none (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11237
2022-05-18T03:36:51.6791206Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11238
2022-05-18T03:36:51.6814671Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11239
2022-05-18T03:36:51.6839724Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11240
2022-05-18T03:36:52.2771424Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0r0xu5ca
2022-05-18T03:36:52.2776847Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0r0xu5ca/_remote_module_non_scriptable.py
2022-05-18T03:36:52.2917192Z dist init r=0, world=4
2022-05-18T03:36:52.3011647Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzqycuuy7
2022-05-18T03:36:52.3012671Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzqycuuy7/_remote_module_non_scriptable.py
2022-05-18T03:36:52.3040854Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpstl5w01t
2022-05-18T03:36:52.3042122Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpstl5w01t/_remote_module_non_scriptable.py
2022-05-18T03:36:52.3135256Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyts66mww
2022-05-18T03:36:52.3137832Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyts66mww/_remote_module_non_scriptable.py
2022-05-18T03:36:52.3183446Z dist init r=3, world=4
2022-05-18T03:36:52.3204464Z dist init r=1, world=4
2022-05-18T03:36:52.3300115Z dist init r=2, world=4
2022-05-18T03:36:52.3511105Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:36:52.3610647Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:36:52.3712508Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:36:52.3713190Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:36:52.3714227Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:52.3714875Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:52.3715709Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:52.3716337Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:52.3822215Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:36:52.3822852Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:36:52.3823531Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:36:52.3824093Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:36:52.5866168Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:36:52.5909927Z   test_nested_wrapped_model_offload_true_none_shard_grad_op (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11293
2022-05-18T03:36:52.5935336Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11294
2022-05-18T03:36:52.5959381Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11295
2022-05-18T03:36:52.5983067Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11296
2022-05-18T03:36:53.1604163Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjlmgyo2i
2022-05-18T03:36:53.1604951Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjlmgyo2i/_remote_module_non_scriptable.py
2022-05-18T03:36:53.1764113Z dist init r=2, world=4
2022-05-18T03:36:53.1844396Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpelit06w7
2022-05-18T03:36:53.1845962Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpelit06w7/_remote_module_non_scriptable.py
2022-05-18T03:36:53.2000460Z dist init r=3, world=4
2022-05-18T03:36:53.2150717Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7er2pcor
2022-05-18T03:36:53.2152561Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7er2pcor/_remote_module_non_scriptable.py
2022-05-18T03:36:53.2256388Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptcev3hn0
2022-05-18T03:36:53.2258571Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptcev3hn0/_remote_module_non_scriptable.py
2022-05-18T03:36:53.2306712Z dist init r=1, world=4
2022-05-18T03:36:53.2411480Z dist init r=0, world=4
2022-05-18T03:36:53.2677783Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:36:53.2779005Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:36:53.2881887Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:36:53.2883205Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:53.2883846Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:36:53.2884546Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:53.2885147Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:53.2885671Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:53.2890672Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:36:53.2891140Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:36:53.2891669Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:36:53.2892211Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:36:53.5009412Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:36:53.5052165Z   test_nested_wrapped_model_offload_true_prefetch_post_no_shard (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11349
2022-05-18T03:36:53.5078465Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11350
2022-05-18T03:36:53.5102074Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11351
2022-05-18T03:36:53.5127126Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11352
2022-05-18T03:36:54.1203915Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy4l_3v1i
2022-05-18T03:36:54.1204654Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy4l_3v1i/_remote_module_non_scriptable.py
2022-05-18T03:36:54.1213893Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfpxi8adx
2022-05-18T03:36:54.1215780Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfpxi8adx/_remote_module_non_scriptable.py
2022-05-18T03:36:54.1362792Z dist init r=0, world=4
2022-05-18T03:36:54.1373082Z dist init r=3, world=4
2022-05-18T03:36:54.1497696Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpys_5hlsf
2022-05-18T03:36:54.1499631Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpys_5hlsf/_remote_module_non_scriptable.py
2022-05-18T03:36:54.1654028Z dist init r=1, world=4
2022-05-18T03:36:54.1658095Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_yojd8u7
2022-05-18T03:36:54.1660180Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_yojd8u7/_remote_module_non_scriptable.py
2022-05-18T03:36:54.1814065Z dist init r=2, world=4
2022-05-18T03:36:54.1984227Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:36:54.2123716Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:36:54.2226699Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:36:54.2227126Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:36:54.2227803Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:54.2228431Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:54.2228960Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:54.2288174Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:54.2333229Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:36:54.2334017Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:36:54.2334934Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:36:54.2335523Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:36:54.4153828Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:36:54.4196726Z   test_nested_wrapped_model_offload_true_prefetch_post_none (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11405
2022-05-18T03:36:54.4223836Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11406
2022-05-18T03:36:54.4248444Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11407
2022-05-18T03:36:54.4273146Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11408
2022-05-18T03:36:55.0147275Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvojjowlk
2022-05-18T03:36:55.0148575Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvojjowlk/_remote_module_non_scriptable.py
2022-05-18T03:36:55.0248827Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdlmwvtz3
2022-05-18T03:36:55.0249887Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdlmwvtz3/_remote_module_non_scriptable.py
2022-05-18T03:36:55.0319941Z dist init r=2, world=4
2022-05-18T03:36:55.0363334Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjgjjptlo
2022-05-18T03:36:55.0364861Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjgjjptlo/_remote_module_non_scriptable.py
2022-05-18T03:36:55.0415786Z dist init r=0, world=4
2022-05-18T03:36:55.0421799Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpffafey7l
2022-05-18T03:36:55.0424668Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpffafey7l/_remote_module_non_scriptable.py
2022-05-18T03:36:55.0521806Z dist init r=1, world=4
2022-05-18T03:36:55.0584408Z dist init r=3, world=4
2022-05-18T03:36:55.0692185Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:36:55.0793010Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:36:55.0894722Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:36:55.0895770Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:36:55.0896583Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:55.0897147Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:55.0897667Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:55.0898180Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:55.1002321Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:36:55.1003001Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:36:55.1003362Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:36:55.1003775Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:36:55.3300233Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:36:55.3343281Z   test_nested_wrapped_model_offload_true_prefetch_post_shard_grad_op (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11461
2022-05-18T03:36:55.3369927Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11462
2022-05-18T03:36:55.3393446Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11463
2022-05-18T03:36:55.3418400Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11464
2022-05-18T03:36:55.9364664Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjgbvla60
2022-05-18T03:36:55.9366129Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjgbvla60/_remote_module_non_scriptable.py
2022-05-18T03:36:55.9460541Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7jqhjow6
2022-05-18T03:36:55.9461742Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7jqhjow6/_remote_module_non_scriptable.py
2022-05-18T03:36:55.9526447Z dist init r=3, world=4
2022-05-18T03:36:55.9551893Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt892mbnn
2022-05-18T03:36:55.9553867Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt892mbnn/_remote_module_non_scriptable.py
2022-05-18T03:36:55.9564247Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjn259vj4
2022-05-18T03:36:55.9566648Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjn259vj4/_remote_module_non_scriptable.py
2022-05-18T03:36:55.9620474Z dist init r=1, world=4
2022-05-18T03:36:55.9712521Z dist init r=0, world=4
2022-05-18T03:36:55.9721999Z dist init r=2, world=4
2022-05-18T03:36:56.0123205Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:36:56.0225993Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:36:56.0226551Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:36:56.0227549Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:56.0228167Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:36:56.0228795Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:56.0229550Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:56.0230137Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:56.0234705Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:36:56.0235310Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:36:56.0235879Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:36:56.0236396Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:36:56.2444569Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:36:56.2485776Z   test_nested_wrapped_model_offload_true_prefetch_pre_no_shard (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11517
2022-05-18T03:36:56.2512724Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11518
2022-05-18T03:36:56.2535754Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11519
2022-05-18T03:36:56.2559649Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11520
2022-05-18T03:36:56.8488691Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzrqdurvj
2022-05-18T03:36:56.8489695Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzrqdurvj/_remote_module_non_scriptable.py
2022-05-18T03:36:56.8610389Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpezk9im07
2022-05-18T03:36:56.8611522Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpezk9im07/_remote_module_non_scriptable.py
2022-05-18T03:36:56.8627237Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxarzi1_s
2022-05-18T03:36:56.8629761Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxarzi1_s/_remote_module_non_scriptable.py
2022-05-18T03:36:56.8647235Z dist init r=1, world=4
2022-05-18T03:36:56.8721064Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsrfz9j3m
2022-05-18T03:36:56.8722683Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsrfz9j3m/_remote_module_non_scriptable.py
2022-05-18T03:36:56.8772243Z dist init r=0, world=4
2022-05-18T03:36:56.8786712Z dist init r=3, world=4
2022-05-18T03:36:56.8879792Z dist init r=2, world=4
2022-05-18T03:36:56.8995932Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:36:56.9097456Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:36:56.9198631Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:36:56.9199195Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:36:56.9200025Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:56.9200831Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:56.9201538Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:56.9202092Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:56.9306342Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:36:56.9307041Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:36:56.9307697Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:36:56.9308293Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:36:57.1586544Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:36:57.1628424Z   test_nested_wrapped_model_offload_true_prefetch_pre_none (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11573
2022-05-18T03:36:57.1655939Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11574
2022-05-18T03:36:57.1679056Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11575
2022-05-18T03:36:57.1703504Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11576
2022-05-18T03:36:57.7571507Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkf14sbrg
2022-05-18T03:36:57.7572829Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkf14sbrg/_remote_module_non_scriptable.py
2022-05-18T03:36:57.7729602Z dist init r=3, world=4
2022-05-18T03:36:57.7730264Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm4562tkv
2022-05-18T03:36:57.7731150Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm4562tkv/_remote_module_non_scriptable.py
2022-05-18T03:36:57.7859010Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4jzvit_w
2022-05-18T03:36:57.7860493Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4jzvit_w/_remote_module_non_scriptable.py
2022-05-18T03:36:57.7889235Z dist init r=0, world=4
2022-05-18T03:36:57.7977554Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5xx_9m_h
2022-05-18T03:36:57.7979092Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5xx_9m_h/_remote_module_non_scriptable.py
2022-05-18T03:36:57.8017996Z dist init r=1, world=4
2022-05-18T03:36:57.8133288Z dist init r=2, world=4
2022-05-18T03:36:57.8339171Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:36:57.8430088Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:36:57.8430499Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:36:57.8431178Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:57.8431595Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:36:57.8432146Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:57.8432661Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:57.8441224Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:57.8537958Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:36:57.8538388Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:36:57.8539151Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:36:58.0731466Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:36:58.0732075Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:36:58.0773687Z   test_nested_wrapped_model_offload_true_prefetch_pre_shard_grad_op (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11629
2022-05-18T03:36:58.0799954Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11630
2022-05-18T03:36:58.0823596Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11631
2022-05-18T03:36:58.0847377Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11632
2022-05-18T03:36:58.6550811Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph1r1uauk
2022-05-18T03:36:58.6551816Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph1r1uauk/_remote_module_non_scriptable.py
2022-05-18T03:36:58.6684329Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc1knk9fi
2022-05-18T03:36:58.6685174Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc1knk9fi/_remote_module_non_scriptable.py
2022-05-18T03:36:58.6710516Z dist init r=3, world=4
2022-05-18T03:36:58.6840981Z dist init r=0, world=4
2022-05-18T03:36:58.7032909Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx5wuvpa2
2022-05-18T03:36:58.7034122Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx5wuvpa2/_remote_module_non_scriptable.py
2022-05-18T03:36:58.7132757Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5td9fz70
2022-05-18T03:36:58.7134728Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5td9fz70/_remote_module_non_scriptable.py
2022-05-18T03:36:58.7188477Z dist init r=1, world=4
2022-05-18T03:36:58.7288783Z dist init r=2, world=4
2022-05-18T03:36:58.7421768Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:36:58.7523213Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:36:58.7599990Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:36:58.7601257Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:58.7601826Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:36:58.7602334Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:58.7625138Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:58.7625826Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:58.7731269Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:36:58.7731722Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:36:58.7732268Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:36:58.7732802Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:36:58.9874560Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:36:58.9918530Z   test_nested_wrapped_model_single_iteration_mixed_precision_cpu_offload_CPUOffload(offload_params=False)_sharding_strategy_None_mixed_precision_False (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11685
2022-05-18T03:36:58.9944779Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11686
2022-05-18T03:36:58.9968201Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11687
2022-05-18T03:36:58.9992534Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11688
2022-05-18T03:36:59.5895822Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1u4w48vl
2022-05-18T03:36:59.5896623Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1u4w48vl/_remote_module_non_scriptable.py
2022-05-18T03:36:59.6015049Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7dvgpvj7
2022-05-18T03:36:59.6015801Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7dvgpvj7/_remote_module_non_scriptable.py
2022-05-18T03:36:59.6052763Z dist init r=2, world=4
2022-05-18T03:36:59.6174997Z dist init r=0, world=4
2022-05-18T03:36:59.6249992Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyrc_dny6
2022-05-18T03:36:59.6252082Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyrc_dny6/_remote_module_non_scriptable.py
2022-05-18T03:36:59.6296039Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbwqh4if1
2022-05-18T03:36:59.6298185Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbwqh4if1/_remote_module_non_scriptable.py
2022-05-18T03:36:59.6408287Z dist init r=3, world=4
2022-05-18T03:36:59.6453668Z dist init r=1, world=4
2022-05-18T03:36:59.6617979Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:36:59.6764867Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:36:59.6765295Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:36:59.6765761Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:36:59.6766490Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:59.6767083Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:59.6767764Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:59.6821169Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:36:59.6873464Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:36:59.6873974Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:36:59.6874530Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:36:59.6875361Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:36:59.9019020Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:36:59.9064241Z   test_nested_wrapped_model_single_iteration_mixed_precision_cpu_offload_CPUOffload(offload_params=False)_sharding_strategy_None_mixed_precision_True (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11741
2022-05-18T03:36:59.9090968Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11742
2022-05-18T03:36:59.9114645Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11743
2022-05-18T03:36:59.9144990Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11744
2022-05-18T03:37:00.4997911Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvfqr8p4y
2022-05-18T03:37:00.4999150Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvfqr8p4y/_remote_module_non_scriptable.py
2022-05-18T03:37:00.5072645Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpru2tvr7x
2022-05-18T03:37:00.5074809Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpru2tvr7x/_remote_module_non_scriptable.py
2022-05-18T03:37:00.5159349Z dist init r=1, world=4
2022-05-18T03:37:00.5233345Z dist init r=0, world=4
2022-05-18T03:37:00.5598907Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0wzd7sz0
2022-05-18T03:37:00.5599690Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0wzd7sz0/_remote_module_non_scriptable.py
2022-05-18T03:37:00.5636685Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4larsiwf
2022-05-18T03:37:00.5638781Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4larsiwf/_remote_module_non_scriptable.py
2022-05-18T03:37:00.5753804Z dist init r=2, world=4
2022-05-18T03:37:00.5794030Z dist init r=3, world=4
2022-05-18T03:37:00.5902442Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:37:00.5972477Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:37:00.6073619Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:37:00.6074367Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:37:00.6075085Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:00.6075600Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:00.6076120Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:00.6105933Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:00.6181465Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:37:00.6182016Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:37:00.6182762Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:37:00.6184846Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:37:00.8171626Z skip: Need at least 2 CUDA devices (0.915s)
2022-05-18T03:37:00.8216176Z   test_nested_wrapped_model_single_iteration_mixed_precision_cpu_offload_CPUOffload(offload_params=False)_sharding_strategy_ShardingStrategy_NO_SHARD_mixed_precision_False (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11797
2022-05-18T03:37:00.8241909Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11798
2022-05-18T03:37:00.8264528Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11799
2022-05-18T03:37:00.8288543Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11800
2022-05-18T03:37:01.4028750Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9d_n7z3z
2022-05-18T03:37:01.4029699Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9d_n7z3z/_remote_module_non_scriptable.py
2022-05-18T03:37:01.4097358Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0496e_ma
2022-05-18T03:37:01.4098860Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0496e_ma/_remote_module_non_scriptable.py
2022-05-18T03:37:01.4189863Z dist init r=1, world=4
2022-05-18T03:37:01.4259754Z dist init r=2, world=4
2022-05-18T03:37:01.4464361Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2c8z4pw1
2022-05-18T03:37:01.4466514Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2c8z4pw1/_remote_module_non_scriptable.py
2022-05-18T03:37:01.4565984Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzq8zpfb8
2022-05-18T03:37:01.4567706Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzq8zpfb8/_remote_module_non_scriptable.py
2022-05-18T03:37:01.4627536Z dist init r=3, world=4
2022-05-18T03:37:01.4723697Z dist init r=0, world=4
2022-05-18T03:37:01.5002491Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:37:01.5105596Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:37:01.5106309Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:37:01.5106826Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:37:01.5107427Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:01.5107960Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:01.5108545Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:01.5205816Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:01.5212896Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:37:01.5213777Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:37:01.5214308Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:37:01.5214851Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:37:01.7315660Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:37:01.7360597Z   test_nested_wrapped_model_single_iteration_mixed_precision_cpu_offload_CPUOffload(offload_params=False)_sharding_strategy_ShardingStrategy_NO_SHARD_mixed_precision_True (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11853
2022-05-18T03:37:01.7386620Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11854
2022-05-18T03:37:01.7409982Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11855
2022-05-18T03:37:01.7434614Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11856
2022-05-18T03:37:02.3443418Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4ra9tqkh
2022-05-18T03:37:02.3444528Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4ra9tqkh/_remote_module_non_scriptable.py
2022-05-18T03:37:02.3504712Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkj0astl8
2022-05-18T03:37:02.3506287Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkj0astl8/_remote_module_non_scriptable.py
2022-05-18T03:37:02.3557160Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpoilw7xio
2022-05-18T03:37:02.3558738Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpoilw7xio/_remote_module_non_scriptable.py
2022-05-18T03:37:02.3578096Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp848dhwgh
2022-05-18T03:37:02.3579601Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp848dhwgh/_remote_module_non_scriptable.py
2022-05-18T03:37:02.3608258Z dist init r=2, world=4
2022-05-18T03:37:02.3676365Z dist init r=1, world=4
2022-05-18T03:37:02.3726436Z dist init r=3, world=4
2022-05-18T03:37:02.3746622Z dist init r=0, world=4
2022-05-18T03:37:02.3934809Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:37:02.4036059Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:37:02.4121035Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:37:02.4121768Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:37:02.4122439Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:02.4122974Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:02.4138225Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:02.4138943Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:02.4230082Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:37:02.4230646Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:37:02.4231159Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:37:02.4231722Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:37:02.6461096Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:37:02.6505426Z   test_nested_wrapped_model_single_iteration_mixed_precision_cpu_offload_CPUOffload(offload_params=False)_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP_mixed_precision_False (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11909
2022-05-18T03:37:02.6530861Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11910
2022-05-18T03:37:02.6554543Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11911
2022-05-18T03:37:02.6578694Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11912
2022-05-18T03:37:03.2872342Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuo_8hemf
2022-05-18T03:37:03.2873989Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuo_8hemf/_remote_module_non_scriptable.py
2022-05-18T03:37:03.2897155Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp70rl3yww
2022-05-18T03:37:03.2898478Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp70rl3yww/_remote_module_non_scriptable.py
2022-05-18T03:37:03.2933269Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz3tn41hs
2022-05-18T03:37:03.2935117Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz3tn41hs/_remote_module_non_scriptable.py
2022-05-18T03:37:03.3033916Z dist init r=2, world=4
2022-05-18T03:37:03.3047724Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwln1bsxs
2022-05-18T03:37:03.3049283Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwln1bsxs/_remote_module_non_scriptable.py
2022-05-18T03:37:03.3060162Z dist init r=0, world=4
2022-05-18T03:37:03.3093709Z dist init r=3, world=4
2022-05-18T03:37:03.3204618Z dist init r=1, world=4
2022-05-18T03:37:03.3445363Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:37:03.3546531Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:37:03.3547263Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:37:03.3547916Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:37:03.3548806Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:03.3549335Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:03.3549864Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:03.3550371Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:03.3653657Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:37:03.3654239Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:37:03.3654789Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:37:03.3655293Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:37:03.5604407Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:37:03.5648562Z   test_nested_wrapped_model_single_iteration_mixed_precision_cpu_offload_CPUOffload(offload_params=False)_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP_mixed_precision_True (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11965
2022-05-18T03:37:03.5675495Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11966
2022-05-18T03:37:03.5699725Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11967
2022-05-18T03:37:03.5724808Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11968
2022-05-18T03:37:04.1572782Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbrokrki7
2022-05-18T03:37:04.1573589Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbrokrki7/_remote_module_non_scriptable.py
2022-05-18T03:37:04.1642187Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_oky9lk9
2022-05-18T03:37:04.1643826Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_oky9lk9/_remote_module_non_scriptable.py
2022-05-18T03:37:04.1732354Z dist init r=0, world=4
2022-05-18T03:37:04.1800030Z dist init r=2, world=4
2022-05-18T03:37:04.1897324Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy5eatz80
2022-05-18T03:37:04.1899207Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy5eatz80/_remote_module_non_scriptable.py
2022-05-18T03:37:04.2060487Z dist init r=3, world=4
2022-05-18T03:37:04.2063075Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8jyirujg
2022-05-18T03:37:04.2065500Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8jyirujg/_remote_module_non_scriptable.py
2022-05-18T03:37:04.2220532Z dist init r=1, world=4
2022-05-18T03:37:04.2512955Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:37:04.2613516Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:37:04.2715894Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:37:04.2716610Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:37:04.2717380Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:04.2717917Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:04.2718440Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:04.2718954Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:04.2824074Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:37:04.2824655Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:37:04.2825180Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:37:04.2825706Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:37:04.4750984Z skip: Need at least 2 CUDA devices (0.915s)
2022-05-18T03:37:04.4795375Z   test_nested_wrapped_model_single_iteration_mixed_precision_cpu_offload_CPUOffload(offload_params=True)_sharding_strategy_None_mixed_precision_False (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12021
2022-05-18T03:37:04.4821472Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12022
2022-05-18T03:37:04.4845237Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12023
2022-05-18T03:37:04.4869791Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 12024
2022-05-18T03:37:05.0723723Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph8qudxge
2022-05-18T03:37:05.0724665Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph8qudxge/_remote_module_non_scriptable.py
2022-05-18T03:37:05.0883947Z dist init r=0, world=4
2022-05-18T03:37:05.0910934Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp07mu53h
2022-05-18T03:37:05.0913290Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp07mu53h/_remote_module_non_scriptable.py
2022-05-18T03:37:05.1074941Z dist init r=2, world=4
2022-05-18T03:37:05.1075454Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwwk24_9i
2022-05-18T03:37:05.1076930Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwwk24_9i/_remote_module_non_scriptable.py
2022-05-18T03:37:05.1234293Z dist init r=1, world=4
2022-05-18T03:37:05.1293905Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp35ady270
2022-05-18T03:37:05.1295781Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp35ady270/_remote_module_non_scriptable.py
2022-05-18T03:37:05.1450592Z dist init r=3, world=4
2022-05-18T03:37:05.1559131Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:37:05.1697756Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:37:05.1799506Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:37:05.1800008Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:37:05.1800639Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:05.1801157Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:05.1801685Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:05.1863800Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:05.1907011Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:37:05.1907749Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:37:05.1908312Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:37:05.1908849Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:37:05.3897440Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:37:05.3943653Z   test_nested_wrapped_model_single_iteration_mixed_precision_cpu_offload_CPUOffload(offload_params=True)_sharding_strategy_None_mixed_precision_True (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12077
2022-05-18T03:37:05.3970447Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12078
2022-05-18T03:37:05.3994434Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12079
2022-05-18T03:37:05.4019127Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 12080
2022-05-18T03:37:05.9891614Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxkxer5l4
2022-05-18T03:37:05.9892426Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxkxer5l4/_remote_module_non_scriptable.py
2022-05-18T03:37:06.0035610Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnbe3h9ox
2022-05-18T03:37:06.0036894Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnbe3h9ox/_remote_module_non_scriptable.py
2022-05-18T03:37:06.0052308Z dist init r=3, world=4
2022-05-18T03:37:06.0205922Z dist init r=2, world=4
2022-05-18T03:37:06.0303458Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2m56tdqy
2022-05-18T03:37:06.0304635Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2m56tdqy/_remote_module_non_scriptable.py
2022-05-18T03:37:06.0377713Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw35cnzr9
2022-05-18T03:37:06.0378644Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw35cnzr9/_remote_module_non_scriptable.py
2022-05-18T03:37:06.0479944Z dist init r=0, world=4
2022-05-18T03:37:06.0550757Z dist init r=1, world=4
2022-05-18T03:37:06.0893350Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:37:06.0893742Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:37:06.0995550Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:37:06.0996129Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:37:06.0996886Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:06.0997593Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:06.0998170Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:06.0998673Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:06.1102414Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:37:06.1103126Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:37:06.1103708Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:37:06.1104239Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:37:06.3045657Z skip: Need at least 2 CUDA devices (0.915s)
2022-05-18T03:37:06.3089336Z   test_nested_wrapped_model_single_iteration_mixed_precision_cpu_offload_CPUOffload(offload_params=True)_sharding_strategy_ShardingStrategy_NO_SHARD_mixed_precision_False (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12133
2022-05-18T03:37:06.3115190Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12134
2022-05-18T03:37:06.3138388Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12135
2022-05-18T03:37:06.3162222Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 12136
2022-05-18T03:37:06.9340083Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzse9wh_h
2022-05-18T03:37:06.9341383Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzse9wh_h/_remote_module_non_scriptable.py
2022-05-18T03:37:06.9351520Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpedctvdo9
2022-05-18T03:37:06.9353856Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpedctvdo9/_remote_module_non_scriptable.py
2022-05-18T03:37:06.9499803Z dist init r=2, world=4
2022-05-18T03:37:06.9510507Z dist init r=1, world=4
2022-05-18T03:37:06.9561903Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6x73ncsy
2022-05-18T03:37:06.9564063Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6x73ncsy/_remote_module_non_scriptable.py
2022-05-18T03:37:06.9719095Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplg_csu6f
2022-05-18T03:37:06.9719579Z dist init r=3, world=4
2022-05-18T03:37:06.9721036Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplg_csu6f/_remote_module_non_scriptable.py
2022-05-18T03:37:06.9873824Z dist init r=0, world=4
2022-05-18T03:37:07.0029001Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:37:07.0130376Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:37:07.0232724Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:37:07.0233222Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:37:07.0233861Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:07.0234378Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:07.0234903Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:07.0333369Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:07.0341272Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:37:07.0342081Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:37:07.0342575Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:37:07.0343103Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:37:07.2188858Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:37:07.2232707Z   test_nested_wrapped_model_single_iteration_mixed_precision_cpu_offload_CPUOffload(offload_params=True)_sharding_strategy_ShardingStrategy_NO_SHARD_mixed_precision_True (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12189
2022-05-18T03:37:07.2258680Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12190
2022-05-18T03:37:07.2281987Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12191
2022-05-18T03:37:07.2305901Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 12192
2022-05-18T03:37:07.7930540Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpepojzdda
2022-05-18T03:37:07.7931799Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpepojzdda/_remote_module_non_scriptable.py
2022-05-18T03:37:07.8053913Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1xlffz1l
2022-05-18T03:37:07.8055227Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1xlffz1l/_remote_module_non_scriptable.py
2022-05-18T03:37:07.8091717Z dist init r=1, world=4
2022-05-18T03:37:07.8213422Z dist init r=2, world=4
2022-05-18T03:37:07.8353873Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgs2bhat_
2022-05-18T03:37:07.8355633Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgs2bhat_/_remote_module_non_scriptable.py
2022-05-18T03:37:07.8509514Z dist init r=0, world=4
2022-05-18T03:37:07.8562105Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_3_kqork
2022-05-18T03:37:07.8564160Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_3_kqork/_remote_module_non_scriptable.py
2022-05-18T03:37:07.8716265Z dist init r=3, world=4
2022-05-18T03:37:07.8904671Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:37:07.8905078Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:37:07.9005889Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:37:07.9006489Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:37:07.9007383Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:07.9008151Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:07.9008892Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:07.9009694Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:07.9016450Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:37:07.9017129Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:37:07.9017707Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:37:07.9018232Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:37:08.1332402Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:37:08.1376468Z   test_nested_wrapped_model_single_iteration_mixed_precision_cpu_offload_CPUOffload(offload_params=True)_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP_mixed_precision_False (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12245
2022-05-18T03:37:08.1402916Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12246
2022-05-18T03:37:08.1425922Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12247
2022-05-18T03:37:08.1450450Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 12248
2022-05-18T03:37:08.7524214Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpro6iqftj
2022-05-18T03:37:08.7524983Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpro6iqftj/_remote_module_non_scriptable.py
2022-05-18T03:37:08.7572940Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1u20vvnd
2022-05-18T03:37:08.7574390Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1u20vvnd/_remote_module_non_scriptable.py
2022-05-18T03:37:08.7578602Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyz16v0j0
2022-05-18T03:37:08.7581002Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyz16v0j0/_remote_module_non_scriptable.py
2022-05-18T03:37:08.7682934Z dist init r=3, world=4
2022-05-18T03:37:08.7703952Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0imtt35h
2022-05-18T03:37:08.7706276Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0imtt35h/_remote_module_non_scriptable.py
2022-05-18T03:37:08.7734172Z dist init r=2, world=4
2022-05-18T03:37:08.7738830Z dist init r=1, world=4
2022-05-18T03:37:08.7863981Z dist init r=0, world=4
2022-05-18T03:37:08.8093681Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:37:08.8094076Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:37:08.8196524Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:37:08.8197855Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:08.8198437Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:37:08.8199116Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:08.8199772Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:08.8200572Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:08.8202584Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:37:08.8203620Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:37:08.8204329Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:37:08.8206083Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:37:09.0475973Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:37:09.0520396Z   test_nested_wrapped_model_single_iteration_mixed_precision_cpu_offload_CPUOffload(offload_params=True)_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP_mixed_precision_True (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12301
2022-05-18T03:37:09.0546006Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12302
2022-05-18T03:37:09.0569568Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12303
2022-05-18T03:37:09.0594036Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 12304
2022-05-18T03:37:09.6571034Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3xiwd2dn
2022-05-18T03:37:09.6572331Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3xiwd2dn/_remote_module_non_scriptable.py
2022-05-18T03:37:09.6613894Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcrp9invh
2022-05-18T03:37:09.6615044Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcrp9invh/_remote_module_non_scriptable.py
2022-05-18T03:37:09.6732902Z dist init r=1, world=4
2022-05-18T03:37:09.6743626Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsr4rysf6
2022-05-18T03:37:09.6745657Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsr4rysf6/_remote_module_non_scriptable.py
2022-05-18T03:37:09.6776327Z dist init r=0, world=4
2022-05-18T03:37:09.6903834Z dist init r=3, world=4
2022-05-18T03:37:09.7012760Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj9mn31r2
2022-05-18T03:37:09.7014610Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj9mn31r2/_remote_module_non_scriptable.py
2022-05-18T03:37:09.7168916Z dist init r=2, world=4
2022-05-18T03:37:09.7314629Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:37:09.7414727Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:37:09.7517149Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:37:09.7517669Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:37:09.7518487Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:09.7519087Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:09.7519624Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:09.7520134Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:09.7625560Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:37:09.7626150Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:37:09.7626862Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:37:09.7627388Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:37:09.9620771Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:37:09.9662854Z   test_transformer_parameterized_offload_false_none_no_shard_clip_norm_type_2_0 (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12357
2022-05-18T03:37:09.9689037Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12358
2022-05-18T03:37:09.9712974Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12359
2022-05-18T03:37:09.9737140Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 12360
2022-05-18T03:37:10.5703242Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9nak_j3u
2022-05-18T03:37:10.5703946Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9nak_j3u/_remote_module_non_scriptable.py
2022-05-18T03:37:10.5730741Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0qd6lpzt
2022-05-18T03:37:10.5732301Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0qd6lpzt/_remote_module_non_scriptable.py
2022-05-18T03:37:10.5783477Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5_prwxys
2022-05-18T03:37:10.5784985Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5_prwxys/_remote_module_non_scriptable.py
2022-05-18T03:37:10.5867306Z dist init r=0, world=4
2022-05-18T03:37:10.5890545Z dist init r=3, world=4
2022-05-18T03:37:10.5940903Z dist init r=2, world=4
2022-05-18T03:37:10.6006822Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9qis1imd
2022-05-18T03:37:10.6008540Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9qis1imd/_remote_module_non_scriptable.py
2022-05-18T03:37:10.6161820Z dist init r=1, world=4
2022-05-18T03:37:10.6453048Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:37:10.6554362Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:37:10.6554979Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:37:10.6555661Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:37:10.6556564Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:10.6557285Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:10.6559423Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:10.6560128Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:10.6563902Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:37:10.6564456Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:37:10.6566260Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:37:10.6566671Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:37:10.8763420Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:37:10.8805140Z   test_transformer_parameterized_offload_false_none_no_shard_clip_norm_type_None (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12413
2022-05-18T03:37:10.8831987Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12414
2022-05-18T03:37:10.8855274Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12415
2022-05-18T03:37:10.8879443Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 12416
2022-05-18T03:37:11.4738780Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph6x6zn5z
2022-05-18T03:37:11.4739557Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph6x6zn5z/_remote_module_non_scriptable.py
2022-05-18T03:37:11.4899344Z dist init r=1, world=4
2022-05-18T03:37:11.5146564Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgrbmrps7
2022-05-18T03:37:11.5147150Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphyaqty3d
2022-05-18T03:37:11.5148203Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgrbmrps7/_remote_module_non_scriptable.py
2022-05-18T03:37:11.5148929Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphyaqty3d/_remote_module_non_scriptable.py
2022-05-18T03:37:11.5248927Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp49u2n3gu
2022-05-18T03:37:11.5250428Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp49u2n3gu/_remote_module_non_scriptable.py
2022-05-18T03:37:11.5307215Z dist init r=3, world=4
2022-05-18T03:37:11.5308782Z dist init r=0, world=4
2022-05-18T03:37:11.5409205Z dist init r=2, world=4
2022-05-18T03:37:11.5619708Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:37:11.5720149Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:37:11.5822093Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:37:11.5822676Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:37:11.5823970Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:11.5824886Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:11.5825592Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:11.5826104Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:11.5830933Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:37:11.5831572Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:37:11.5832156Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:37:11.5832785Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:37:11.7906511Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:37:11.7948470Z   test_transformer_parameterized_offload_false_none_none_clip_norm_type_2_0 (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12469
2022-05-18T03:37:11.7974661Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12470
2022-05-18T03:37:11.7998092Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12471
2022-05-18T03:37:11.8021513Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 12472
2022-05-18T03:37:12.3673603Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuma3d73p
2022-05-18T03:37:12.3674329Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuma3d73p/_remote_module_non_scriptable.py
2022-05-18T03:37:12.3829715Z dist init r=2, world=4
2022-05-18T03:37:12.3936795Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5rbtrzm1
2022-05-18T03:37:12.3937989Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5rbtrzm1/_remote_module_non_scriptable.py
2022-05-18T03:37:12.4093216Z dist init r=3, world=4
2022-05-18T03:37:12.4170359Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpviq5h3ed
2022-05-18T03:37:12.4172162Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpviq5h3ed/_remote_module_non_scriptable.py
2022-05-18T03:37:12.4327673Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvbpcxaht
2022-05-18T03:37:12.4328307Z dist init r=0, world=4
2022-05-18T03:37:12.4329586Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvbpcxaht/_remote_module_non_scriptable.py
2022-05-18T03:37:12.4484180Z dist init r=1, world=4
2022-05-18T03:37:12.4706151Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:37:12.4907707Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:37:12.5009931Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:37:12.5011008Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:12.5011567Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:37:12.5012064Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:12.5012811Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:12.5013338Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:12.5117471Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:37:12.5118007Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:37:12.5118573Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:37:12.5119106Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:37:12.7048234Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:37:12.7089750Z   test_transformer_parameterized_offload_false_none_none_clip_norm_type_None (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12525
2022-05-18T03:37:12.7116020Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12526
2022-05-18T03:37:12.7138676Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12527
2022-05-18T03:37:12.7162351Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 12528
2022-05-18T03:37:13.3223157Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw7nxy9gc
2022-05-18T03:37:13.3224644Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw7nxy9gc/_remote_module_non_scriptable.py
2022-05-18T03:37:13.3376681Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3__trgzq
2022-05-18T03:37:13.3378423Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3__trgzq/_remote_module_non_scriptable.py
2022-05-18T03:37:13.3384311Z dist init r=2, world=4
2022-05-18T03:37:13.3465081Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgwod_0ln
2022-05-18T03:37:13.3466850Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgwod_0ln/_remote_module_non_scriptable.py
2022-05-18T03:37:13.3538625Z dist init r=3, world=4
2022-05-18T03:37:13.3624308Z dist init r=1, world=4
2022-05-18T03:37:13.3670407Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfefpr4f5
2022-05-18T03:37:13.3672415Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfefpr4f5/_remote_module_non_scriptable.py
2022-05-18T03:37:13.3828965Z dist init r=0, world=4
2022-05-18T03:37:13.4238003Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:37:13.4338970Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:37:13.4440600Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:37:13.4441265Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:37:13.4442407Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:13.4443039Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:13.4443550Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:13.4444066Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:13.4549262Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:37:13.4549830Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:37:13.4550631Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:37:13.4551149Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:37:13.6189520Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:37:13.6231503Z   test_transformer_parameterized_offload_false_none_shard_grad_op_clip_norm_type_2_0 (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12581
2022-05-18T03:37:13.6257430Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12582
2022-05-18T03:37:13.6280611Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12583
2022-05-18T03:37:13.6304312Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 12584
2022-05-18T03:37:14.2314701Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp39t6_19b
2022-05-18T03:37:14.2315632Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp39t6_19b/_remote_module_non_scriptable.py
2022-05-18T03:37:14.2367045Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc9f0_m14
2022-05-18T03:37:14.2368501Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc9f0_m14/_remote_module_non_scriptable.py
2022-05-18T03:37:14.2380860Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6vpscjzi
2022-05-18T03:37:14.2383393Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6vpscjzi/_remote_module_non_scriptable.py
2022-05-18T03:37:14.2476388Z dist init r=1, world=4
2022-05-18T03:37:14.2526699Z dist init r=2, world=4
2022-05-18T03:37:14.2540241Z dist init r=3, world=4
2022-05-18T03:37:14.2778528Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr1t5xp31
2022-05-18T03:37:14.2779459Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr1t5xp31/_remote_module_non_scriptable.py
2022-05-18T03:37:14.2935264Z dist init r=0, world=4
2022-05-18T03:37:14.3289242Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:37:14.3289662Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:37:14.3392214Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:37:14.3393228Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:14.3393947Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:37:14.3394879Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:14.3395709Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:14.3396477Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:14.3400502Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:37:14.3401077Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:37:14.3401607Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:37:14.3402142Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:37:14.5331990Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:37:14.5374605Z   test_transformer_parameterized_offload_false_none_shard_grad_op_clip_norm_type_None (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12637
2022-05-18T03:37:14.5400020Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12638
2022-05-18T03:37:14.5423467Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12639
2022-05-18T03:37:14.5447527Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 12640
2022-05-18T03:37:15.1411488Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxx5wuzpt
2022-05-18T03:37:15.1412257Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxx5wuzpt/_remote_module_non_scriptable.py
2022-05-18T03:37:15.1505533Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7hlz3icn
2022-05-18T03:37:15.1506368Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7hlz3icn/_remote_module_non_scriptable.py
2022-05-18T03:37:15.1571659Z dist init r=3, world=4
2022-05-18T03:37:15.1664092Z dist init r=2, world=4
2022-05-18T03:37:15.1682307Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwy54eue3
2022-05-18T03:37:15.1682738Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb3a88_ad
2022-05-18T03:37:15.1685074Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwy54eue3/_remote_module_non_scriptable.py
2022-05-18T03:37:15.1685582Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb3a88_ad/_remote_module_non_scriptable.py
2022-05-18T03:37:15.1842327Z dist init r=0, world=4
2022-05-18T03:37:15.1843705Z dist init r=1, world=4
2022-05-18T03:37:15.2157239Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:37:15.2257482Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:37:15.2360298Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:37:15.2361296Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:15.2361936Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:37:15.2362486Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:15.2363166Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:15.2363687Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:15.2368069Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:37:15.2368580Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:37:15.2368959Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:37:15.2369481Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:37:15.4474657Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:37:15.4517170Z   test_transformer_parameterized_offload_false_prefetch_post_no_shard_clip_norm_type_2_0 (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12693
2022-05-18T03:37:15.4543929Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12694
2022-05-18T03:37:15.4566981Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12695
2022-05-18T03:37:15.4591223Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 12696
2022-05-18T03:37:16.0561479Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph2hoo2aq
2022-05-18T03:37:16.0562227Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph2hoo2aq/_remote_module_non_scriptable.py
2022-05-18T03:37:16.0637057Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqwz0yfar
2022-05-18T03:37:16.0638438Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqwz0yfar/_remote_module_non_scriptable.py
2022-05-18T03:37:16.0673151Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr6c4815k
2022-05-18T03:37:16.0675608Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr6c4815k/_remote_module_non_scriptable.py
2022-05-18T03:37:16.0720639Z dist init r=0, world=4
2022-05-18T03:37:16.0722882Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2bwndt2v
2022-05-18T03:37:16.0725405Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2bwndt2v/_remote_module_non_scriptable.py
2022-05-18T03:37:16.0799425Z dist init r=3, world=4
2022-05-18T03:37:16.0830508Z dist init r=1, world=4
2022-05-18T03:37:16.0882128Z dist init r=2, world=4
2022-05-18T03:37:16.1141141Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:37:16.1241961Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:37:16.1344435Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:37:16.1345056Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:37:16.1345849Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:16.1346366Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:16.1346897Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:16.1347415Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:16.1450591Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:37:16.1451356Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:37:16.1451943Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:37:16.1452385Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:37:16.3617923Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:37:16.3659664Z   test_transformer_parameterized_offload_false_prefetch_post_no_shard_clip_norm_type_None (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12749
2022-05-18T03:37:16.3686283Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12750
2022-05-18T03:37:16.3709566Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12751
2022-05-18T03:37:16.3733576Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 12752
2022-05-18T03:37:16.9544593Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg71_6bf0
2022-05-18T03:37:16.9545594Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg71_6bf0/_remote_module_non_scriptable.py
2022-05-18T03:37:16.9707401Z dist init r=1, world=4
2022-05-18T03:37:16.9838688Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpov12kz0w
2022-05-18T03:37:16.9839907Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy1dr1lsu
2022-05-18T03:37:16.9840338Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpov12kz0w/_remote_module_non_scriptable.py
2022-05-18T03:37:16.9840788Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy1dr1lsu/_remote_module_non_scriptable.py
2022-05-18T03:37:16.9994593Z dist init r=3, world=4
2022-05-18T03:37:16.9994914Z dist init r=2, world=4
2022-05-18T03:37:17.0005155Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp416zkfxu
2022-05-18T03:37:17.0007033Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp416zkfxu/_remote_module_non_scriptable.py
2022-05-18T03:37:17.0161291Z dist init r=0, world=4
2022-05-18T03:37:17.0519680Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:37:17.0621243Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:37:17.0723196Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:37:17.0723877Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:37:17.0724660Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:17.0725176Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:17.0725699Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:17.0823936Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:17.0830479Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:37:17.0831131Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:37:17.0831787Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:37:17.0832353Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:37:17.2761088Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:37:17.2803676Z   test_transformer_parameterized_offload_false_prefetch_post_none_clip_norm_type_2_0 (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12805
2022-05-18T03:37:17.2830306Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12806
2022-05-18T03:37:17.2854288Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12807
2022-05-18T03:37:17.2879373Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 12808
2022-05-18T03:37:17.8771341Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpybdvrhv0
2022-05-18T03:37:17.8772609Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpybdvrhv0/_remote_module_non_scriptable.py
2022-05-18T03:37:17.8928139Z dist init r=0, world=4
2022-05-18T03:37:17.9035676Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_727jh17
2022-05-18T03:37:17.9037928Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_727jh17/_remote_module_non_scriptable.py
2022-05-18T03:37:17.9046916Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxmcxw5qt
2022-05-18T03:37:17.9048933Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxmcxw5qt/_remote_module_non_scriptable.py
2022-05-18T03:37:17.9157955Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdgt5d5q7
2022-05-18T03:37:17.9165910Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdgt5d5q7/_remote_module_non_scriptable.py
2022-05-18T03:37:17.9193850Z dist init r=3, world=4
2022-05-18T03:37:17.9204482Z dist init r=2, world=4
2022-05-18T03:37:17.9318957Z dist init r=1, world=4
2022-05-18T03:37:17.9513357Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:37:17.9629903Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:37:17.9630379Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:37:17.9630790Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:37:17.9631572Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:17.9632159Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:17.9632679Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:17.9716692Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:17.9737135Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:37:17.9737838Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:37:17.9738388Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:37:17.9739035Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:37:18.1905585Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:37:18.1947806Z   test_transformer_parameterized_offload_false_prefetch_post_none_clip_norm_type_None (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12861
2022-05-18T03:37:18.1973626Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12862
2022-05-18T03:37:18.1996524Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12863
2022-05-18T03:37:18.2020812Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 12864
2022-05-18T03:37:18.7681888Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6lo9gte6
2022-05-18T03:37:18.7682871Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6lo9gte6/_remote_module_non_scriptable.py
2022-05-18T03:37:18.7840853Z dist init r=3, world=4
2022-05-18T03:37:18.8165477Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqpgl32oq
2022-05-18T03:37:18.8166357Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqpgl32oq/_remote_module_non_scriptable.py
2022-05-18T03:37:18.8219209Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_s5wbacd
2022-05-18T03:37:18.8221239Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_s5wbacd/_remote_module_non_scriptable.py
2022-05-18T03:37:18.8323523Z dist init r=1, world=4
2022-05-18T03:37:18.8374874Z dist init r=2, world=4
2022-05-18T03:37:18.8393591Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdg814d0v
2022-05-18T03:37:18.8395731Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdg814d0v/_remote_module_non_scriptable.py
2022-05-18T03:37:18.8549277Z dist init r=0, world=4
2022-05-18T03:37:18.8751679Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:37:18.8853170Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:37:18.8955305Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:37:18.8956226Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:18.8956814Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:37:18.8957440Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:18.8958072Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:18.8958872Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:18.8962927Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:37:18.8963794Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:37:18.8964338Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:37:18.8964906Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:37:19.1047914Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:37:19.1089885Z   test_transformer_parameterized_offload_false_prefetch_post_shard_grad_op_clip_norm_type_2_0 (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12917
2022-05-18T03:37:19.1116494Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12918
2022-05-18T03:37:19.1139723Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12919
2022-05-18T03:37:19.1164392Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 12920
2022-05-18T03:37:19.7140432Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprspgwuov
2022-05-18T03:37:19.7141206Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprspgwuov/_remote_module_non_scriptable.py
2022-05-18T03:37:19.7293379Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpasm3765w
2022-05-18T03:37:19.7294205Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpasm3765w/_remote_module_non_scriptable.py
2022-05-18T03:37:19.7305024Z dist init r=1, world=4
2022-05-18T03:37:19.7451906Z dist init r=2, world=4
2022-05-18T03:37:19.7603211Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvy2qtgpl
2022-05-18T03:37:19.7604471Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvy2qtgpl/_remote_module_non_scriptable.py
2022-05-18T03:37:19.7609409Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdskx51fh
2022-05-18T03:37:19.7611923Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdskx51fh/_remote_module_non_scriptable.py
2022-05-18T03:37:19.7761047Z dist init r=0, world=4
2022-05-18T03:37:19.7769504Z dist init r=3, world=4
2022-05-18T03:37:19.8064428Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:37:19.8164915Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:37:19.8165688Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:19.8166126Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:37:19.8166820Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:19.8167403Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:37:19.8168919Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:19.8169474Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:19.8271977Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:37:19.8272534Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:37:19.8273063Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:37:19.8273594Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:37:20.0191022Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:37:20.0233488Z   test_transformer_parameterized_offload_false_prefetch_post_shard_grad_op_clip_norm_type_None (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12973
2022-05-18T03:37:20.0260236Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12974
2022-05-18T03:37:20.0284214Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12975
2022-05-18T03:37:20.0308681Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 12976
2022-05-18T03:37:20.5944196Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm6502h66
2022-05-18T03:37:20.5944966Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm6502h66/_remote_module_non_scriptable.py
2022-05-18T03:37:20.6064153Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgapv6mk7
2022-05-18T03:37:20.6065051Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgapv6mk7/_remote_module_non_scriptable.py
2022-05-18T03:37:20.6102782Z dist init r=0, world=4
2022-05-18T03:37:20.6223726Z dist init r=3, world=4
2022-05-18T03:37:20.6505248Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvnwv60t5
2022-05-18T03:37:20.6506092Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvnwv60t5/_remote_module_non_scriptable.py
2022-05-18T03:37:20.6544551Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4_lw_9n5
2022-05-18T03:37:20.6546905Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4_lw_9n5/_remote_module_non_scriptable.py
2022-05-18T03:37:20.6660343Z dist init r=1, world=4
2022-05-18T03:37:20.6701469Z dist init r=2, world=4
2022-05-18T03:37:20.6835690Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:37:20.7012917Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:37:20.7114850Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:37:20.7115707Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:37:20.7116407Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:20.7116936Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:20.7117457Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:20.7139058Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:20.7222784Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:37:20.7223558Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:37:20.7224056Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:37:20.7224423Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:37:20.9335757Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:37:20.9378615Z   test_transformer_parameterized_offload_false_prefetch_pre_no_shard_clip_norm_type_2_0 (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13029
2022-05-18T03:37:20.9405610Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13030
2022-05-18T03:37:20.9428977Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13031
2022-05-18T03:37:20.9453350Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 13032
2022-05-18T03:37:21.5254699Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw7ygvp03
2022-05-18T03:37:21.5255702Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw7ygvp03/_remote_module_non_scriptable.py
2022-05-18T03:37:21.5413835Z dist init r=2, world=4
2022-05-18T03:37:21.5601568Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4pogymtl
2022-05-18T03:37:21.5602311Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4pogymtl/_remote_module_non_scriptable.py
2022-05-18T03:37:21.5648923Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzauw7u3v
2022-05-18T03:37:21.5650467Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzauw7u3v/_remote_module_non_scriptable.py
2022-05-18T03:37:21.5761722Z dist init r=3, world=4
2022-05-18T03:37:21.5809215Z dist init r=0, world=4
2022-05-18T03:37:21.5900663Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgb817n_m
2022-05-18T03:37:21.5902606Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgb817n_m/_remote_module_non_scriptable.py
2022-05-18T03:37:21.6056220Z dist init r=1, world=4
2022-05-18T03:37:21.6226133Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:37:21.6326370Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:37:21.6428924Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:37:21.6429942Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:21.6430506Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:37:21.6431355Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:21.6431878Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:21.6432411Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:21.6535815Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:37:21.6536328Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:37:21.6536852Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:37:21.6537519Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:37:21.8480223Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:37:21.8523268Z   test_transformer_parameterized_offload_false_prefetch_pre_no_shard_clip_norm_type_None (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13085
2022-05-18T03:37:21.8550160Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13086
2022-05-18T03:37:21.8573725Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13087
2022-05-18T03:37:21.8598186Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 13088
2022-05-18T03:37:22.4886134Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplofsp0ly
2022-05-18T03:37:22.4886926Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplofsp0ly/_remote_module_non_scriptable.py
2022-05-18T03:37:22.4937865Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn7kkq0g5
2022-05-18T03:37:22.4939345Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn7kkq0g5/_remote_module_non_scriptable.py
2022-05-18T03:37:22.5047135Z dist init r=0, world=4
2022-05-18T03:37:22.5099708Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7oa93omj
2022-05-18T03:37:22.5100107Z dist init r=3, world=4
2022-05-18T03:37:22.5100483Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7oa93omj/_remote_module_non_scriptable.py
2022-05-18T03:37:22.5257241Z dist init r=1, world=4
2022-05-18T03:37:22.5262357Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpemwfq0e7
2022-05-18T03:37:22.5264572Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpemwfq0e7/_remote_module_non_scriptable.py
2022-05-18T03:37:22.5418559Z dist init r=2, world=4
2022-05-18T03:37:22.5660460Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:37:22.5661249Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:37:22.5661752Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:37:22.5662223Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:37:22.5663089Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:22.5663637Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:22.5665897Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:22.5666552Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:22.5767636Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:37:22.5768246Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:37:22.5768787Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:37:22.5769335Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:37:22.7624166Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:37:22.7665860Z   test_transformer_parameterized_offload_false_prefetch_pre_none_clip_norm_type_2_0 (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13141
2022-05-18T03:37:22.7692757Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13142
2022-05-18T03:37:22.7715518Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13143
2022-05-18T03:37:22.7739467Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 13144
2022-05-18T03:37:23.3765083Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnnh7_j4z
2022-05-18T03:37:23.3766715Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnnh7_j4z/_remote_module_non_scriptable.py
2022-05-18T03:37:23.3844433Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp900j0ldx
2022-05-18T03:37:23.3845895Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp900j0ldx/_remote_module_non_scriptable.py
2022-05-18T03:37:23.3925209Z dist init r=3, world=4
2022-05-18T03:37:23.3966338Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvhowd41k
2022-05-18T03:37:23.3968157Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvhowd41k/_remote_module_non_scriptable.py
2022-05-18T03:37:23.4001734Z dist init r=2, world=4
2022-05-18T03:37:23.4123490Z dist init r=1, world=4
2022-05-18T03:37:23.4169699Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpemxkv_5f
2022-05-18T03:37:23.4172059Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpemxkv_5f/_remote_module_non_scriptable.py
2022-05-18T03:37:23.4329229Z dist init r=0, world=4
2022-05-18T03:37:23.4535694Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:37:23.4715123Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:37:23.4715917Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:37:23.4716418Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:37:23.4717105Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:23.4717620Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:23.4718148Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:23.4738714Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:23.4823377Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:37:23.4823952Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:37:23.4824514Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:37:23.4825074Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:37:23.6768262Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:37:23.6813214Z   test_transformer_parameterized_offload_false_prefetch_pre_none_clip_norm_type_None (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13197
2022-05-18T03:37:23.6839366Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13198
2022-05-18T03:37:23.6862660Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13199
2022-05-18T03:37:23.6887099Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 13200
2022-05-18T03:37:24.3171421Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg_18taoc
2022-05-18T03:37:24.3172192Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg_18taoc/_remote_module_non_scriptable.py
2022-05-18T03:37:24.3330170Z dist init r=2, world=4
2022-05-18T03:37:24.3338172Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg48jzdml
2022-05-18T03:37:24.3339367Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmperam6s0f
2022-05-18T03:37:24.3340017Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg48jzdml/_remote_module_non_scriptable.py
2022-05-18T03:37:24.3342231Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmperam6s0f/_remote_module_non_scriptable.py
2022-05-18T03:37:24.3432106Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9hwjenr5
2022-05-18T03:37:24.3434487Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9hwjenr5/_remote_module_non_scriptable.py
2022-05-18T03:37:24.3502435Z dist init r=0, world=4
2022-05-18T03:37:24.3504070Z dist init r=3, world=4
2022-05-18T03:37:24.3591247Z dist init r=1, world=4
2022-05-18T03:37:24.3840583Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:37:24.3942177Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:37:24.3943061Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:37:24.3943744Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:37:24.3944567Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:24.3945097Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:24.3947033Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:24.3947997Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:24.4050835Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:37:24.4051367Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:37:24.4051804Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:37:24.4052329Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:37:24.5913102Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:37:24.5954806Z   test_transformer_parameterized_offload_false_prefetch_pre_shard_grad_op_clip_norm_type_2_0 (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13253
2022-05-18T03:37:24.5981017Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13254
2022-05-18T03:37:24.6004383Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13255
2022-05-18T03:37:24.6028516Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 13256
2022-05-18T03:37:25.2230209Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpobei9d6e
2022-05-18T03:37:25.2230969Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpobei9d6e/_remote_module_non_scriptable.py
2022-05-18T03:37:25.2322224Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxe0cemw_
2022-05-18T03:37:25.2323805Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxe0cemw_/_remote_module_non_scriptable.py
2022-05-18T03:37:25.2390844Z dist init r=3, world=4
2022-05-18T03:37:25.2482626Z dist init r=1, world=4
2022-05-18T03:37:25.2585926Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpedgwllil
2022-05-18T03:37:25.2587932Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpedgwllil/_remote_module_non_scriptable.py
2022-05-18T03:37:25.2675890Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7toqngyf
2022-05-18T03:37:25.2677819Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7toqngyf/_remote_module_non_scriptable.py
2022-05-18T03:37:25.2746292Z dist init r=2, world=4
2022-05-18T03:37:25.2834049Z dist init r=0, world=4
2022-05-18T03:37:25.3057519Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:37:25.3157915Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:37:25.3260773Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:37:25.3261301Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:37:25.3261991Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:25.3262529Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:25.3263226Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:25.3263751Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:25.3367540Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:37:25.3368096Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:37:25.3368604Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:37:25.3369347Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:37:25.5054412Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:37:25.5097293Z   test_transformer_parameterized_offload_false_prefetch_pre_shard_grad_op_clip_norm_type_None (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13309
2022-05-18T03:37:25.5122918Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13310
2022-05-18T03:37:25.5146385Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13311
2022-05-18T03:37:25.5170573Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 13312
2022-05-18T03:37:26.0858978Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnb467gc5
2022-05-18T03:37:26.0860498Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnb467gc5/_remote_module_non_scriptable.py
2022-05-18T03:37:26.0907962Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzetge17f
2022-05-18T03:37:26.0909463Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzetge17f/_remote_module_non_scriptable.py
2022-05-18T03:37:26.1017434Z dist init r=0, world=4
2022-05-18T03:37:26.1066598Z dist init r=3, world=4
2022-05-18T03:37:26.1274229Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc84mucdr
2022-05-18T03:37:26.1275054Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc84mucdr/_remote_module_non_scriptable.py
2022-05-18T03:37:26.1378221Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwufjx95c
2022-05-18T03:37:26.1379773Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwufjx95c/_remote_module_non_scriptable.py
2022-05-18T03:37:26.1428899Z dist init r=2, world=4
2022-05-18T03:37:26.1533681Z dist init r=1, world=4
2022-05-18T03:37:26.1931252Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:37:26.2032070Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:37:26.2134742Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:37:26.2135824Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:37:26.2136853Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:26.2137393Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:26.2137903Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:26.2138419Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:26.2244811Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:37:26.2245237Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:37:26.2245578Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:37:26.2245920Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:37:26.4197775Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:37:26.4240885Z   test_transformer_parameterized_offload_true_none_no_shard_clip_norm_type_2_0 (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13365
2022-05-18T03:37:26.4266941Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13366
2022-05-18T03:37:26.4290962Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13367
2022-05-18T03:37:26.4314933Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 13368
2022-05-18T03:37:27.0105670Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6ogs5e88
2022-05-18T03:37:27.0106688Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6ogs5e88/_remote_module_non_scriptable.py
2022-05-18T03:37:27.0227999Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphvzuwl4e
2022-05-18T03:37:27.0229043Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphvzuwl4e/_remote_module_non_scriptable.py
2022-05-18T03:37:27.0266357Z dist init r=1, world=4
2022-05-18T03:37:27.0304787Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmbgf5pl4
2022-05-18T03:37:27.0306901Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmbgf5pl4/_remote_module_non_scriptable.py
2022-05-18T03:37:27.0386974Z dist init r=2, world=4
2022-05-18T03:37:27.0461476Z dist init r=0, world=4
2022-05-18T03:37:27.0498454Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbpkzbc8o
2022-05-18T03:37:27.0500448Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbpkzbc8o/_remote_module_non_scriptable.py
2022-05-18T03:37:27.0654207Z dist init r=3, world=4
2022-05-18T03:37:27.0761570Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:37:27.0799228Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:37:27.0800199Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:27.0800642Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:37:27.0800999Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:37:27.0801504Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:27.0802050Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:27.0864139Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:27.0907334Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:37:27.0907761Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:37:27.0908243Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:37:27.0908787Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:37:27.2340320Z skip: Need at least 2 CUDA devices (0.814s)
2022-05-18T03:37:27.2382434Z   test_transformer_parameterized_offload_true_none_no_shard_clip_norm_type_None (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13421
2022-05-18T03:37:27.2409427Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13422
2022-05-18T03:37:27.2432998Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13423
2022-05-18T03:37:27.2457012Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 13424
2022-05-18T03:37:27.8098866Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp81ef5isq
2022-05-18T03:37:27.8099921Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp81ef5isq/_remote_module_non_scriptable.py
2022-05-18T03:37:27.8256931Z dist init r=3, world=4
2022-05-18T03:37:27.8294807Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyc51xnwd
2022-05-18T03:37:27.8297197Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyc51xnwd/_remote_module_non_scriptable.py
2022-05-18T03:37:27.8451413Z dist init r=1, world=4
2022-05-18T03:37:27.8647350Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_vp3rf5d
2022-05-18T03:37:27.8649299Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_vp3rf5d/_remote_module_non_scriptable.py
2022-05-18T03:37:27.8735567Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpaufl7bto
2022-05-18T03:37:27.8737515Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpaufl7bto/_remote_module_non_scriptable.py
2022-05-18T03:37:27.8804025Z dist init r=2, world=4
2022-05-18T03:37:27.8892078Z dist init r=0, world=4
2022-05-18T03:37:27.9068049Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:37:27.9169406Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:37:27.9271358Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:37:27.9272686Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:37:27.9273558Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:27.9274386Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:27.9275084Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:27.9275651Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:27.9279573Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:37:27.9280466Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:37:27.9281660Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:37:27.9282386Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:37:28.1482834Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:37:28.1525840Z   test_transformer_parameterized_offload_true_none_none_clip_norm_type_2_0 (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13477
2022-05-18T03:37:28.1552107Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13478
2022-05-18T03:37:28.1575515Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13479
2022-05-18T03:37:28.1599960Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 13480
2022-05-18T03:37:28.7236725Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdge1ix05
2022-05-18T03:37:28.7237409Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyfllunri
2022-05-18T03:37:28.7239010Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdge1ix05/_remote_module_non_scriptable.py
2022-05-18T03:37:28.7239739Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyfllunri/_remote_module_non_scriptable.py
2022-05-18T03:37:28.7296068Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv_yzuqig
2022-05-18T03:37:28.7297192Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv_yzuqig/_remote_module_non_scriptable.py
2022-05-18T03:37:28.7396397Z dist init r=3, world=4
2022-05-18T03:37:28.7396742Z dist init r=1, world=4
2022-05-18T03:37:28.7454334Z dist init r=0, world=4
2022-05-18T03:37:28.7784351Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzme79jc7
2022-05-18T03:37:28.7786307Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzme79jc7/_remote_module_non_scriptable.py
2022-05-18T03:37:28.7940705Z dist init r=2, world=4
2022-05-18T03:37:28.8107427Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:37:28.8208635Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:37:28.8310006Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:37:28.8310885Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:37:28.8311568Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:28.8312383Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:28.8312930Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:28.8313431Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:28.8417704Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:37:28.8418286Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:37:28.8418837Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:37:28.8419439Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:37:29.0627404Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:37:29.0668961Z   test_transformer_parameterized_offload_true_none_none_clip_norm_type_None (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13533
2022-05-18T03:37:29.0695687Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13534
2022-05-18T03:37:29.0719395Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13535
2022-05-18T03:37:29.0743062Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 13536
2022-05-18T03:37:29.6314927Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu4n18q1z
2022-05-18T03:37:29.6316505Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu4n18q1z/_remote_module_non_scriptable.py
2022-05-18T03:37:29.6472231Z dist init r=0, world=4
2022-05-18T03:37:29.6636913Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppi9rfuf_
2022-05-18T03:37:29.6638555Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppi9rfuf_/_remote_module_non_scriptable.py
2022-05-18T03:37:29.6793067Z dist init r=1, world=4
2022-05-18T03:37:29.6966698Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa9lc2hdn
2022-05-18T03:37:29.6968596Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa9lc2hdn/_remote_module_non_scriptable.py
2022-05-18T03:37:29.7030333Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6axg_4to
2022-05-18T03:37:29.7031898Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6axg_4to/_remote_module_non_scriptable.py
2022-05-18T03:37:29.7124373Z dist init r=3, world=4
2022-05-18T03:37:29.7188697Z dist init r=2, world=4
2022-05-18T03:37:29.7398006Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:37:29.7485528Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:37:29.7486498Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:29.7487037Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:37:29.7487538Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:37:29.7488516Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:29.7489159Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:29.7500755Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:29.7594326Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:37:29.7594913Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:37:29.7595466Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:37:29.7596012Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:37:29.9770032Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:37:29.9813078Z   test_transformer_parameterized_offload_true_none_shard_grad_op_clip_norm_type_2_0 (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13589
2022-05-18T03:37:29.9839919Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13590
2022-05-18T03:37:29.9863210Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13591
2022-05-18T03:37:29.9887392Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 13592
2022-05-18T03:37:30.5538589Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1q6ica4v
2022-05-18T03:37:30.5539722Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1q6ica4v/_remote_module_non_scriptable.py
2022-05-18T03:37:30.5697885Z dist init r=1, world=4
2022-05-18T03:37:30.6028393Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfp4583gi
2022-05-18T03:37:30.6029563Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfp4583gi/_remote_module_non_scriptable.py
2022-05-18T03:37:30.6184492Z dist init r=3, world=4
2022-05-18T03:37:30.6341640Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4b25qbva
2022-05-18T03:37:30.6343986Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4b25qbva/_remote_module_non_scriptable.py
2022-05-18T03:37:30.6498157Z dist init r=0, world=4
2022-05-18T03:37:30.6837847Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgw3n3ub4
2022-05-18T03:37:30.6839380Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgw3n3ub4/_remote_module_non_scriptable.py
2022-05-18T03:37:30.6990191Z dist init r=2, world=4
2022-05-18T03:37:30.7097923Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:37:30.7210904Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:37:30.7212248Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:30.7212975Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:37:30.7213841Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:37:30.7214945Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:30.7215981Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:30.7301263Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:30.7319114Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:37:30.7319805Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:37:30.7320686Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:37:30.7321361Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:37:30.8913709Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:37:30.8955226Z   test_transformer_parameterized_offload_true_none_shard_grad_op_clip_norm_type_None (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13645
2022-05-18T03:37:30.8980775Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13646
2022-05-18T03:37:30.9004555Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13647
2022-05-18T03:37:30.9028328Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 13648
2022-05-18T03:37:31.4567844Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5jmdbfgr
2022-05-18T03:37:31.4568632Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5jmdbfgr/_remote_module_non_scriptable.py
2022-05-18T03:37:31.4623070Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4ugh87al
2022-05-18T03:37:31.4625567Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4ugh87al/_remote_module_non_scriptable.py
2022-05-18T03:37:31.4665867Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsou7v099
2022-05-18T03:37:31.4668471Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsou7v099/_remote_module_non_scriptable.py
2022-05-18T03:37:31.4733568Z dist init r=2, world=4
2022-05-18T03:37:31.4782671Z dist init r=3, world=4
2022-05-18T03:37:31.4822355Z dist init r=0, world=4
2022-05-18T03:37:31.5175386Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphmv229mt
2022-05-18T03:37:31.5176205Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphmv229mt/_remote_module_non_scriptable.py
2022-05-18T03:37:31.5330372Z dist init r=1, world=4
2022-05-18T03:37:31.5495079Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:37:31.5595787Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:37:31.5697926Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:37:31.5698566Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:37:31.5699472Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:31.5699991Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:31.5700684Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:31.5701310Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:31.5706022Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:37:31.5706413Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:37:31.5706767Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:37:31.5707221Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:37:31.8055157Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:37:31.8097273Z   test_transformer_parameterized_offload_true_prefetch_post_no_shard_clip_norm_type_2_0 (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13701
2022-05-18T03:37:31.8122828Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13702
2022-05-18T03:37:31.8146144Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13703
2022-05-18T03:37:31.8170170Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 13704
2022-05-18T03:37:32.4232128Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk9s0c1_9
2022-05-18T03:37:32.4233490Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk9s0c1_9/_remote_module_non_scriptable.py
2022-05-18T03:37:32.4297151Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb0b5b4tl
2022-05-18T03:37:32.4298254Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb0b5b4tl/_remote_module_non_scriptable.py
2022-05-18T03:37:32.4337808Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpacbiwe6z
2022-05-18T03:37:32.4339912Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpacbiwe6z/_remote_module_non_scriptable.py
2022-05-18T03:37:32.4394629Z dist init r=3, world=4
2022-05-18T03:37:32.4455048Z dist init r=1, world=4
2022-05-18T03:37:32.4496138Z dist init r=0, world=4
2022-05-18T03:37:32.4584585Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpufs9ogis
2022-05-18T03:37:32.4586805Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpufs9ogis/_remote_module_non_scriptable.py
2022-05-18T03:37:32.4739479Z dist init r=2, world=4
2022-05-18T03:37:32.4967349Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:37:32.5068132Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:37:32.5170542Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:37:32.5171344Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:37:32.5172328Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:32.5172961Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:32.5173543Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:32.5174101Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:32.5277366Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:37:32.5277967Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:37:32.5278728Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:37:32.5279121Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:37:32.7196251Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:37:32.7238440Z   test_transformer_parameterized_offload_true_prefetch_post_no_shard_clip_norm_type_None (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13757
2022-05-18T03:37:32.7264111Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13758
2022-05-18T03:37:32.7287540Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13759
2022-05-18T03:37:32.7311724Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 13760
2022-05-18T03:37:33.2921670Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxagw4xve
2022-05-18T03:37:33.2923492Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxagw4xve/_remote_module_non_scriptable.py
2022-05-18T03:37:33.2924805Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgm3asna7
2022-05-18T03:37:33.2927875Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgm3asna7/_remote_module_non_scriptable.py
2022-05-18T03:37:33.2940054Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpym9r32l2
2022-05-18T03:37:33.2942500Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpym9r32l2/_remote_module_non_scriptable.py
2022-05-18T03:37:33.3083459Z dist init r=2, world=4
2022-05-18T03:37:33.3086183Z dist init r=1, world=4
2022-05-18T03:37:33.3099089Z dist init r=3, world=4
2022-05-18T03:37:33.3537301Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxyoyg45p
2022-05-18T03:37:33.3538205Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxyoyg45p/_remote_module_non_scriptable.py
2022-05-18T03:37:33.3690631Z dist init r=0, world=4
2022-05-18T03:37:33.4101996Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:37:33.4102552Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:37:33.4103268Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:37:33.4104258Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:33.4104767Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:37:33.4105559Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:33.4106241Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:33.4106991Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:33.4111207Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:37:33.4111801Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:37:33.4112362Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:37:33.4112778Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:37:33.6338390Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:37:33.6381105Z   test_transformer_parameterized_offload_true_prefetch_post_none_clip_norm_type_2_0 (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13813
2022-05-18T03:37:33.6407105Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13814
2022-05-18T03:37:33.6430914Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13815
2022-05-18T03:37:33.6454604Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 13816
2022-05-18T03:37:34.2307532Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbm5lltai
2022-05-18T03:37:34.2308396Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbm5lltai/_remote_module_non_scriptable.py
2022-05-18T03:37:34.2436349Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf60sgad3
2022-05-18T03:37:34.2437314Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf60sgad3/_remote_module_non_scriptable.py
2022-05-18T03:37:34.2468020Z dist init r=0, world=4
2022-05-18T03:37:34.2601166Z dist init r=3, world=4
2022-05-18T03:37:34.2807765Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw_sv2yy1
2022-05-18T03:37:34.2809277Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw_sv2yy1/_remote_module_non_scriptable.py
2022-05-18T03:37:34.2961594Z dist init r=1, world=4
2022-05-18T03:37:34.2969375Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgdkycwnd
2022-05-18T03:37:34.2971438Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgdkycwnd/_remote_module_non_scriptable.py
2022-05-18T03:37:34.3124985Z dist init r=2, world=4
2022-05-18T03:37:34.3313231Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:37:34.3434418Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:37:34.3536233Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:37:34.3536891Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:37:34.3537904Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:34.3538447Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:34.3538984Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:34.3617508Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:34.3644132Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:37:34.3644690Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:37:34.3645237Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:37:34.3645771Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:37:34.5480794Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:37:34.5522560Z   test_transformer_parameterized_offload_true_prefetch_post_none_clip_norm_type_None (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13869
2022-05-18T03:37:34.5547397Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13870
2022-05-18T03:37:34.5570313Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13871
2022-05-18T03:37:34.5594165Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 13872
2022-05-18T03:37:35.1666743Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv8pgntni
2022-05-18T03:37:35.1668015Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv8pgntni/_remote_module_non_scriptable.py
2022-05-18T03:37:35.1764029Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmposjom0zk
2022-05-18T03:37:35.1766209Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmposjom0zk/_remote_module_non_scriptable.py
2022-05-18T03:37:35.1791790Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwbrr5lj6
2022-05-18T03:37:35.1794120Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwbrr5lj6/_remote_module_non_scriptable.py
2022-05-18T03:37:35.1799977Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp17mfs0eh
2022-05-18T03:37:35.1801788Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp17mfs0eh/_remote_module_non_scriptable.py
2022-05-18T03:37:35.1831474Z dist init r=3, world=4
2022-05-18T03:37:35.1926964Z dist init r=1, world=4
2022-05-18T03:37:35.1951320Z dist init r=0, world=4
2022-05-18T03:37:35.1957624Z dist init r=2, world=4
2022-05-18T03:37:35.2140028Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:37:35.2241008Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:37:35.2344018Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:37:35.2345059Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:35.2346036Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:37:35.2346758Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:35.2347499Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:35.2348413Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:35.2351083Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:37:35.2351663Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:37:35.2352308Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:37:35.2352878Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:37:35.4620388Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:37:35.4662684Z   test_transformer_parameterized_offload_true_prefetch_post_shard_grad_op_clip_norm_type_2_0 (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13925
2022-05-18T03:37:35.4688944Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13926
2022-05-18T03:37:35.4711867Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13927
2022-05-18T03:37:35.4735611Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 13928
2022-05-18T03:37:36.0401044Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfx4zyr5h
2022-05-18T03:37:36.0401767Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfx4zyr5h/_remote_module_non_scriptable.py
2022-05-18T03:37:36.0559946Z dist init r=3, world=4
2022-05-18T03:37:36.0590145Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplo0ie_gm
2022-05-18T03:37:36.0591663Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplo0ie_gm/_remote_module_non_scriptable.py
2022-05-18T03:37:36.0647749Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdyxzxibm
2022-05-18T03:37:36.0649576Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdyxzxibm/_remote_module_non_scriptable.py
2022-05-18T03:37:36.0749720Z dist init r=1, world=4
2022-05-18T03:37:36.0806011Z dist init r=0, world=4
2022-05-18T03:37:36.0968282Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjai7ekar
2022-05-18T03:37:36.0969243Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjai7ekar/_remote_module_non_scriptable.py
2022-05-18T03:37:36.1123115Z dist init r=2, world=4
2022-05-18T03:37:36.1269603Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:37:36.1434353Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:37:36.1434766Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:37:36.1435396Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:36.1435801Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:37:36.1436295Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:36.1436802Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:36.1472734Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:36.1541694Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:37:36.1542431Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:37:36.1543142Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:37:36.1543710Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:37:36.3762604Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:37:36.3805700Z   test_transformer_parameterized_offload_true_prefetch_post_shard_grad_op_clip_norm_type_None (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13981
2022-05-18T03:37:36.3832257Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13982
2022-05-18T03:37:36.3855485Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13983
2022-05-18T03:37:36.3878850Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 13984
2022-05-18T03:37:37.0180965Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5dwa_m3c
2022-05-18T03:37:37.0182145Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5dwa_m3c/_remote_module_non_scriptable.py
2022-05-18T03:37:37.0184501Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc0k73th0
2022-05-18T03:37:37.0186666Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc0k73th0/_remote_module_non_scriptable.py
2022-05-18T03:37:37.0210750Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz2urze1z
2022-05-18T03:37:37.0212262Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz2urze1z/_remote_module_non_scriptable.py
2022-05-18T03:37:37.0339608Z dist init r=2, world=4
2022-05-18T03:37:37.0347374Z dist init r=0, world=4
2022-05-18T03:37:37.0369594Z dist init r=3, world=4
2022-05-18T03:37:37.0374920Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppxyxlrdg
2022-05-18T03:37:37.0377043Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppxyxlrdg/_remote_module_non_scriptable.py
2022-05-18T03:37:37.0529798Z dist init r=1, world=4
2022-05-18T03:37:37.0679254Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:37:37.0780268Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:37:37.0881453Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:37:37.0882025Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:37:37.0882867Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:37.0883577Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:37.0884114Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:37.0884624Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:37.0990302Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:37:37.0990996Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:37:37.0991529Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:37:37.0992067Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:37:37.2906028Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:37:37.2951747Z   test_transformer_parameterized_offload_true_prefetch_pre_no_shard_clip_norm_type_2_0 (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14037
2022-05-18T03:37:37.2979190Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14038
2022-05-18T03:37:37.3003186Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14039
2022-05-18T03:37:37.3027455Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 14040
2022-05-18T03:37:37.9169910Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf_9r21zj
2022-05-18T03:37:37.9170688Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf_9r21zj/_remote_module_non_scriptable.py
2022-05-18T03:37:37.9291910Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwz2opwo7
2022-05-18T03:37:37.9292998Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwz2opwo7/_remote_module_non_scriptable.py
2022-05-18T03:37:37.9333117Z dist init r=3, world=4
2022-05-18T03:37:37.9398697Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp856uk2o7
2022-05-18T03:37:37.9400005Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp856uk2o7/_remote_module_non_scriptable.py
2022-05-18T03:37:37.9430263Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0qylb0ai
2022-05-18T03:37:37.9432237Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0qylb0ai/_remote_module_non_scriptable.py
2022-05-18T03:37:37.9454591Z dist init r=2, world=4
2022-05-18T03:37:37.9559461Z dist init r=0, world=4
2022-05-18T03:37:37.9588256Z dist init r=1, world=4
2022-05-18T03:37:37.9844395Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:37:38.0047595Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:37:38.0048272Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:37:38.0049013Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:38.0049421Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:37:38.0049917Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:38.0050423Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:38.0148246Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:38.0155029Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:37:38.0155707Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:37:38.0156252Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:37:38.0157620Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:37:38.2054061Z skip: Need at least 2 CUDA devices (0.915s)
2022-05-18T03:37:38.2097891Z   test_transformer_parameterized_offload_true_prefetch_pre_no_shard_clip_norm_type_None (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14093
2022-05-18T03:37:38.2123879Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14094
2022-05-18T03:37:38.2146922Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14095
2022-05-18T03:37:38.2170648Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 14096
2022-05-18T03:37:38.8183350Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7j6jyp18
2022-05-18T03:37:38.8184319Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7j6jyp18/_remote_module_non_scriptable.py
2022-05-18T03:37:38.8341596Z dist init r=2, world=4
2022-05-18T03:37:38.8504100Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_e2k5t68
2022-05-18T03:37:38.8505211Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_e2k5t68/_remote_module_non_scriptable.py
2022-05-18T03:37:38.8637737Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3j7ntweh
2022-05-18T03:37:38.8640264Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3j7ntweh/_remote_module_non_scriptable.py
2022-05-18T03:37:38.8640902Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph47u50rp
2022-05-18T03:37:38.8643283Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph47u50rp/_remote_module_non_scriptable.py
2022-05-18T03:37:38.8665995Z dist init r=1, world=4
2022-05-18T03:37:38.8795051Z dist init r=3, world=4
2022-05-18T03:37:38.8799501Z dist init r=0, world=4
2022-05-18T03:37:38.9178667Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:37:38.9280204Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:37:38.9281077Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:37:38.9281987Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:37:38.9282957Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:38.9283578Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:38.9284093Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:38.9284628Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:38.9387806Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:37:38.9388484Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:37:38.9388911Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:37:38.9389759Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:37:39.1197584Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:37:39.1240235Z   test_transformer_parameterized_offload_true_prefetch_pre_none_clip_norm_type_2_0 (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14149
2022-05-18T03:37:39.1266525Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14150
2022-05-18T03:37:39.1290285Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14151
2022-05-18T03:37:39.1314248Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 14152
2022-05-18T03:37:39.7276465Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj88budxd
2022-05-18T03:37:39.7277250Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj88budxd/_remote_module_non_scriptable.py
2022-05-18T03:37:39.7379958Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp915kdpif
2022-05-18T03:37:39.7383064Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp915kdpif/_remote_module_non_scriptable.py
2022-05-18T03:37:39.7440380Z dist init r=2, world=4
2022-05-18T03:37:39.7537143Z dist init r=3, world=4
2022-05-18T03:37:39.7580868Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzrldtoki
2022-05-18T03:37:39.7583677Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzrldtoki/_remote_module_non_scriptable.py
2022-05-18T03:37:39.7589232Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf7dqj5tw
2022-05-18T03:37:39.7591163Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf7dqj5tw/_remote_module_non_scriptable.py
2022-05-18T03:37:39.7738073Z dist init r=0, world=4
2022-05-18T03:37:39.7749874Z dist init r=1, world=4
2022-05-18T03:37:39.8048093Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:37:39.8048492Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:37:39.8150015Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:37:39.8150619Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:37:39.8151556Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:39.8152309Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:39.8152945Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:39.8153768Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:39.8157959Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:37:39.8158538Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:37:39.8160488Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:37:39.8161120Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:37:40.0341400Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:37:40.0386255Z   test_transformer_parameterized_offload_true_prefetch_pre_none_clip_norm_type_None (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14205
2022-05-18T03:37:40.0412762Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14206
2022-05-18T03:37:40.0437017Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14207
2022-05-18T03:37:40.0461746Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 14208
2022-05-18T03:37:40.6511439Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0af5on0e
2022-05-18T03:37:40.6512421Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0af5on0e/_remote_module_non_scriptable.py
2022-05-18T03:37:40.6670503Z dist init r=0, world=4
2022-05-18T03:37:40.6723674Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbr_n7h2i
2022-05-18T03:37:40.6724965Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbr_n7h2i/_remote_module_non_scriptable.py
2022-05-18T03:37:40.6838151Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwbq6wrcg
2022-05-18T03:37:40.6839553Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwbq6wrcg/_remote_module_non_scriptable.py
2022-05-18T03:37:40.6878266Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptmr_768c
2022-05-18T03:37:40.6879509Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptmr_768c/_remote_module_non_scriptable.py
2022-05-18T03:37:40.6882799Z dist init r=3, world=4
2022-05-18T03:37:40.6996214Z dist init r=2, world=4
2022-05-18T03:37:40.7034614Z dist init r=1, world=4
2022-05-18T03:37:40.7192881Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:37:40.7293667Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:37:40.7395067Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:37:40.7395819Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:37:40.7396729Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:40.7397529Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:40.7398209Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:40.7398779Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:40.7503360Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:37:40.7504071Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:37:40.7504618Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:37:40.7505161Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:37:40.9488237Z skip: Need at least 2 CUDA devices (0.915s)
2022-05-18T03:37:40.9530393Z   test_transformer_parameterized_offload_true_prefetch_pre_shard_grad_op_clip_norm_type_2_0 (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14261
2022-05-18T03:37:40.9556134Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14262
2022-05-18T03:37:40.9579661Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14263
2022-05-18T03:37:40.9603412Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 14264
2022-05-18T03:37:41.5683795Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7b6it2ub
2022-05-18T03:37:41.5686030Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7b6it2ub/_remote_module_non_scriptable.py
2022-05-18T03:37:41.5686714Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpegrmbx9p
2022-05-18T03:37:41.5688746Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpegrmbx9p/_remote_module_non_scriptable.py
2022-05-18T03:37:41.5718968Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_kpmqnng
2022-05-18T03:37:41.5720189Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_kpmqnng/_remote_module_non_scriptable.py
2022-05-18T03:37:41.5845749Z dist init r=3, world=4
2022-05-18T03:37:41.5846301Z dist init r=0, world=4
2022-05-18T03:37:41.5877640Z dist init r=2, world=4
2022-05-18T03:37:41.6016476Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvu7b8sv1
2022-05-18T03:37:41.6018639Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvu7b8sv1/_remote_module_non_scriptable.py
2022-05-18T03:37:41.6170190Z dist init r=1, world=4
2022-05-18T03:37:41.6480740Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:37:41.6581612Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:37:41.6684629Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:37:41.6685832Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:41.6686586Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:37:41.6687330Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:41.6688018Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:41.6688606Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:41.6695618Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:37:41.6696194Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:37:41.6696742Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:37:41.6699159Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:37:41.8630554Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:37:41.8673021Z   test_transformer_parameterized_offload_true_prefetch_pre_shard_grad_op_clip_norm_type_None (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14317
2022-05-18T03:37:41.8699245Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14318
2022-05-18T03:37:41.8723226Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14319
2022-05-18T03:37:41.8746976Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 14320
2022-05-18T03:37:42.4791261Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm3jucig1
2022-05-18T03:37:42.4792034Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm3jucig1/_remote_module_non_scriptable.py
2022-05-18T03:37:42.4946608Z dist init r=3, world=4
2022-05-18T03:37:42.5102429Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptdurgsob
2022-05-18T03:37:42.5103757Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptdurgsob/_remote_module_non_scriptable.py
2022-05-18T03:37:42.5174433Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8fimyph0
2022-05-18T03:37:42.5176481Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8fimyph0/_remote_module_non_scriptable.py
2022-05-18T03:37:42.5265597Z dist init r=0, world=4
2022-05-18T03:37:42.5266705Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkqyv58a5
2022-05-18T03:37:42.5269532Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkqyv58a5/_remote_module_non_scriptable.py
2022-05-18T03:37:42.5332273Z dist init r=1, world=4
2022-05-18T03:37:42.5424084Z dist init r=2, world=4
2022-05-18T03:37:42.5642680Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:37:42.5744781Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:37:42.5745391Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:37:42.5746420Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:42.5747188Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:37:42.5748174Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:42.5749042Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:42.5749576Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:42.5852409Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:37:42.5852988Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:37:42.5853545Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:37:42.5854150Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:37:42.7773592Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:37:42.7774244Z 
2022-05-18T03:37:42.7775022Z ----------------------------------------------------------------------
2022-05-18T03:37:42.7775563Z Ran 203 tests in 186.106s
2022-05-18T03:37:42.7775732Z 
2022-05-18T03:37:42.7775821Z OK (skipped=203)
2022-05-18T03:37:42.7775931Z 
2022-05-18T03:37:42.7776015Z Generating XML reports...
2022-05-18T03:37:42.7816604Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_core/TEST-TestHooks-20220518033436.xml
2022-05-18T03:37:42.7819852Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_core/TEST-TestNoGrad-20220518033436.xml
2022-05-18T03:37:42.7823594Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_core/TEST-TestParamInit-20220518033436.xml
2022-05-18T03:37:42.8437093Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_core/TEST-TestParityWithDDP-20220518033436.xml
2022-05-18T03:37:43.0136534Z Running distributed/fsdp/test_fsdp_exec_order ... [2022-05-18 03:37:43.013262]
2022-05-18T03:37:43.0137346Z Executing ['/opt/conda/bin/python', 'distributed/fsdp/test_fsdp_exec_order.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-05-18 03:37:43.013347]
2022-05-18T03:37:43.5850753Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_exec_order
2022-05-18T03:37:43.5869057Z 
2022-05-18T03:37:43.5869153Z Running tests...
2022-05-18T03:37:43.5869862Z ----------------------------------------------------------------------
2022-05-18T03:37:43.5877677Z   test_invalid_first_iter_order_sharding_strategy_ShardingStrategy_FULL_SHARD (__main__.TestFSDPExecOrder)
2022-05-18T03:37:43.8649185Z Tests that FSDP errors if the all-gather order differs across ranks ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14384
2022-05-18T03:37:43.8670479Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14385
2022-05-18T03:37:43.8692723Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14386
2022-05-18T03:37:43.8716000Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 14387
2022-05-18T03:37:44.5348342Z dist init r=0, world=4
2022-05-18T03:37:44.5464991Z dist init r=3, world=4
2022-05-18T03:37:44.6020808Z dist init r=1, world=4
2022-05-18T03:37:44.6077643Z dist init r=2, world=4
2022-05-18T03:37:44.6287172Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:37:44.6388410Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:37:44.6489755Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:37:44.6491042Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:44.6491541Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:37:44.6492171Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:44.6492702Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:44.6493225Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:44.6500536Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:37:44.6501392Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:37:44.6502024Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:37:44.6502866Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:37:44.8748007Z skip: Need at least 2 CUDA devices (1.288s)
2022-05-18T03:37:44.8755423Z   test_invalid_first_iter_order_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestFSDPExecOrder)
2022-05-18T03:37:44.8793100Z Tests that FSDP errors if the all-gather order differs across ranks ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14440
2022-05-18T03:37:44.8817642Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14441
2022-05-18T03:37:44.8840663Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14442
2022-05-18T03:37:44.8864620Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 14443
2022-05-18T03:37:45.4803139Z dist init r=1, world=4
2022-05-18T03:37:45.4813808Z dist init r=3, world=4
2022-05-18T03:37:45.5060573Z dist init r=0, world=4
2022-05-18T03:37:45.5270514Z dist init r=2, world=4
2022-05-18T03:37:45.5479437Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:37:45.5515291Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:37:45.5616698Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:37:45.5617346Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:37:45.5618224Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:45.5618982Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:45.5619506Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:45.5683295Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:45.5724536Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:37:45.5725216Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:37:45.5725599Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:37:45.5725941Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:37:45.7890411Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:37:45.7904614Z   test_invalid_later_iter_order_sharding_strategy_ShardingStrategy_FULL_SHARD_iters_before_path_change_1 (__main__.TestFSDPExecOrder)
2022-05-18T03:37:45.7940699Z Tests that FSDP warns the user if the all-gather order changes after ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14496
2022-05-18T03:37:45.7967094Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14497
2022-05-18T03:37:45.7990049Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14498
2022-05-18T03:37:45.8014282Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 14499
2022-05-18T03:37:46.3747186Z dist init r=1, world=4
2022-05-18T03:37:46.3912511Z dist init r=2, world=4
2022-05-18T03:37:46.4254951Z dist init r=0, world=4
2022-05-18T03:37:46.4484803Z dist init r=3, world=4
2022-05-18T03:37:46.4592190Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:37:46.4659959Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:37:46.4660816Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:46.4661327Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:37:46.4661983Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:37:46.4662576Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:46.4663342Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:46.4694618Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:46.4768596Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:37:46.4769239Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:37:46.4769595Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:37:46.4769939Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:37:46.7040423Z skip: Need at least 2 CUDA devices (0.915s)
2022-05-18T03:37:46.7054058Z   test_invalid_later_iter_order_sharding_strategy_ShardingStrategy_FULL_SHARD_iters_before_path_change_3 (__main__.TestFSDPExecOrder)
2022-05-18T03:37:46.7090342Z Tests that FSDP warns the user if the all-gather order changes after ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14552
2022-05-18T03:37:46.7116448Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14553
2022-05-18T03:37:46.7139984Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14554
2022-05-18T03:37:46.7164238Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 14555
2022-05-18T03:37:47.2882084Z dist init r=1, world=4
2022-05-18T03:37:47.3300038Z dist init r=3, world=4
2022-05-18T03:37:47.3469851Z dist init r=2, world=4
2022-05-18T03:37:47.3524854Z dist init r=0, world=4
2022-05-18T03:37:47.3896209Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:37:47.3896623Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:37:47.3998649Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:47.3999218Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:37:47.3999822Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:37:47.4000763Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:47.4001515Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:47.4002278Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:47.4006699Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:37:47.4007413Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:37:47.4007924Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:37:47.4008502Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:37:47.6191245Z skip: Need at least 2 CUDA devices (0.915s)
2022-05-18T03:37:47.6204709Z   test_invalid_later_iter_order_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP_iters_before_path_change_1 (__main__.TestFSDPExecOrder)
2022-05-18T03:37:47.6241289Z Tests that FSDP warns the user if the all-gather order changes after ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14608
2022-05-18T03:37:47.6267843Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14609
2022-05-18T03:37:47.6291609Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14610
2022-05-18T03:37:47.6315661Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 14611
2022-05-18T03:37:48.2125420Z dist init r=3, world=4
2022-05-18T03:37:48.2208445Z dist init r=1, world=4
2022-05-18T03:37:48.2629871Z dist init r=2, world=4
2022-05-18T03:37:48.2630253Z dist init r=0, world=4
2022-05-18T03:37:48.2836106Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:37:48.2938556Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:37:48.2939298Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:37:48.2939882Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:37:48.2940605Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:48.2941517Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:48.3039640Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:48.3040214Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:48.3047964Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:37:48.3048597Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:37:48.3049161Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:37:48.3049688Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:37:48.5342310Z skip: Need at least 2 CUDA devices (0.915s)
2022-05-18T03:37:48.5355968Z   test_invalid_later_iter_order_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP_iters_before_path_change_3 (__main__.TestFSDPExecOrder)
2022-05-18T03:37:48.5394127Z Tests that FSDP warns the user if the all-gather order changes after ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14664
2022-05-18T03:37:48.5419837Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14665
2022-05-18T03:37:48.5443900Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14666
2022-05-18T03:37:48.5468155Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 14667
2022-05-18T03:37:49.1147212Z dist init r=3, world=4
2022-05-18T03:37:49.1505871Z dist init r=1, world=4
2022-05-18T03:37:49.1718245Z dist init r=2, world=4
2022-05-18T03:37:49.1867422Z dist init r=0, world=4
2022-05-18T03:37:49.2058943Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:37:49.2159752Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:37:49.2262333Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:37:49.2263764Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:49.2264343Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:37:49.2264855Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:49.2265380Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:49.2266080Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:49.2273184Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:37:49.2273780Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:37:49.2274398Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:37:49.2274965Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:37:49.4494475Z skip: Need at least 2 CUDA devices (0.915s)
2022-05-18T03:37:49.4542461Z   test_train_eval_sharding_strategy_ShardingStrategy_FULL_SHARD (__main__.TestFSDPExecOrder) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14720
2022-05-18T03:37:49.4569264Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14721
2022-05-18T03:37:49.4592395Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14722
2022-05-18T03:37:49.4616338Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 14723
2022-05-18T03:37:50.0389184Z dist init r=3, world=4
2022-05-18T03:37:50.0420846Z dist init r=1, world=4
2022-05-18T03:37:50.0510824Z dist init r=2, world=4
2022-05-18T03:37:50.1045411Z dist init r=0, world=4
2022-05-18T03:37:50.1334097Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:37:50.1435630Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:37:50.1536434Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:37:50.1537706Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:50.1538520Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:50.1539122Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:37:50.1539796Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:50.1540305Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:50.1643502Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:37:50.1643896Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:37:50.1644255Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:37:50.1644590Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:37:50.3641904Z skip: Need at least 2 CUDA devices (0.915s)
2022-05-18T03:37:50.3691091Z   test_train_eval_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestFSDPExecOrder) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14776
2022-05-18T03:37:50.3717139Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14777
2022-05-18T03:37:50.3741017Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14778
2022-05-18T03:37:50.3765406Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 14779
2022-05-18T03:37:50.9739708Z dist init r=1, world=4
2022-05-18T03:37:50.9952369Z dist init r=3, world=4
2022-05-18T03:37:51.0046734Z dist init r=0, world=4
2022-05-18T03:37:51.0173707Z dist init r=2, world=4
2022-05-18T03:37:51.0452471Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:37:51.0452876Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:37:51.0553604Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:37:51.0554688Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:37:51.0555947Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:51.0556768Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:51.0557613Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:51.0558238Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:51.0561048Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:37:51.0561608Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:37:51.0562146Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:37:51.0562672Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:37:51.2791990Z skip: Need at least 2 CUDA devices (0.915s)
2022-05-18T03:37:51.2792257Z 
2022-05-18T03:37:51.2792717Z ----------------------------------------------------------------------
2022-05-18T03:37:51.2793153Z Ran 8 tests in 7.692s
2022-05-18T03:37:51.2793339Z 
2022-05-18T03:37:51.2793456Z OK (skipped=8)
2022-05-18T03:37:51.2793624Z 
2022-05-18T03:37:51.2793764Z Generating XML reports...
2022-05-18T03:37:51.2834175Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_exec_order/TEST-TestFSDPExecOrder-20220518033743.xml
2022-05-18T03:37:51.4763644Z Running distributed/fsdp/test_fsdp_freezing_weights ... [2022-05-18 03:37:51.476000]
2022-05-18T03:37:51.4764260Z Executing ['/opt/conda/bin/python', 'distributed/fsdp/test_fsdp_freezing_weights.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-05-18 03:37:51.476079]
2022-05-18T03:37:52.0529767Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_freezing_weights
2022-05-18T03:37:52.0547869Z 
2022-05-18T03:37:52.0548196Z Running tests...
2022-05-18T03:37:52.0548838Z ----------------------------------------------------------------------
2022-05-18T03:37:52.3421724Z   test_freezing_weights_with_nested_trunk_False_freezing_method_FreezingMethod_GradToNone_freeze_after_wrap_fsdp_False (__main__.TestFreezingWeights) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14843
2022-05-18T03:37:52.3443122Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14844
2022-05-18T03:37:52.3466808Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14845
2022-05-18T03:37:52.3491058Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 14846
2022-05-18T03:37:52.9648342Z dist init r=2, world=4
2022-05-18T03:37:52.9771048Z dist init r=1, world=4
2022-05-18T03:37:52.9774061Z dist init r=0, world=4
2022-05-18T03:37:53.0006093Z dist init r=3, world=4
2022-05-18T03:37:53.0158972Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:37:53.0259481Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:37:53.0361151Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:37:53.0362402Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:53.0363027Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:37:53.0363790Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:53.0364782Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:53.0365432Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:53.0369849Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:37:53.0371660Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:37:53.0372210Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:37:53.0372768Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:37:53.2522071Z skip: Need at least 2 CUDA devices (1.197s)
2022-05-18T03:37:53.2566603Z   test_freezing_weights_with_nested_trunk_False_freezing_method_FreezingMethod_GradToNone_freeze_after_wrap_fsdp_True (__main__.TestFreezingWeights) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14899
2022-05-18T03:37:53.2593142Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14900
2022-05-18T03:37:53.2616878Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14901
2022-05-18T03:37:53.2640907Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 14902
2022-05-18T03:37:53.8373251Z dist init r=0, world=4
2022-05-18T03:37:53.8393531Z dist init r=1, world=4
2022-05-18T03:37:53.8850695Z dist init r=2, world=4
2022-05-18T03:37:53.9052933Z dist init r=3, world=4
2022-05-18T03:37:53.9160777Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:37:53.9186687Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:37:53.9187227Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:37:53.9187749Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:37:53.9188398Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:53.9188921Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:53.9189446Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:53.9263647Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:53.9294177Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:37:53.9294881Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:37:53.9295396Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:37:53.9295933Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:37:54.1667905Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:37:54.1712800Z   test_freezing_weights_with_nested_trunk_False_freezing_method_FreezingMethod_RequiresGrad_freeze_after_wrap_fsdp_False (__main__.TestFreezingWeights) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14955
2022-05-18T03:37:54.1744292Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14956
2022-05-18T03:37:54.1772894Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14957
2022-05-18T03:37:54.1801064Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 14958
2022-05-18T03:37:54.7592316Z dist init r=0, world=4
2022-05-18T03:37:54.7592889Z dist init r=3, world=4
2022-05-18T03:37:54.7796631Z dist init r=1, world=4
2022-05-18T03:37:54.8171959Z dist init r=2, world=4
2022-05-18T03:37:54.8302455Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:37:54.8403165Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:37:54.8482044Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:37:54.8482775Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:37:54.8483732Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:54.8484258Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:54.8506011Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:54.8506622Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:54.8612264Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:37:54.8612876Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:37:54.8613484Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:37:54.8614009Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:37:55.0827379Z skip: Need at least 2 CUDA devices (0.916s)
2022-05-18T03:37:55.0871693Z   test_freezing_weights_with_nested_trunk_False_freezing_method_FreezingMethod_RequiresGrad_freeze_after_wrap_fsdp_True (__main__.TestFreezingWeights) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15011
2022-05-18T03:37:55.0898297Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15012
2022-05-18T03:37:55.0922018Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15013
2022-05-18T03:37:55.0946460Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 15014
2022-05-18T03:37:55.6708926Z dist init r=3, world=4
2022-05-18T03:37:55.7145454Z dist init r=2, world=4
2022-05-18T03:37:55.7258068Z dist init r=0, world=4
2022-05-18T03:37:55.7276945Z dist init r=1, world=4
2022-05-18T03:37:55.7453811Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:37:55.7521544Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:37:55.7622717Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:37:55.7623449Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:37:55.7624316Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:55.7624865Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:55.7625386Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:55.7657127Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:55.7731618Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:37:55.7732143Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:37:55.7732737Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:37:55.7733519Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:37:55.9972990Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:37:56.0017495Z   test_freezing_weights_with_nested_trunk_True_freezing_method_FreezingMethod_GradToNone_freeze_after_wrap_fsdp_False (__main__.TestFreezingWeights) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15067
2022-05-18T03:37:56.0043555Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15068
2022-05-18T03:37:56.0066790Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15069
2022-05-18T03:37:56.0090645Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 15070
2022-05-18T03:37:56.5897396Z dist init r=2, world=4
2022-05-18T03:37:56.5987261Z dist init r=0, world=4
2022-05-18T03:37:56.6123720Z dist init r=3, world=4
2022-05-18T03:37:56.6455185Z dist init r=1, world=4
2022-05-18T03:37:56.6708991Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:37:56.6809826Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:37:56.6912522Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:37:56.6913126Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:37:56.6913943Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:56.6914567Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:56.6915130Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:56.6915649Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:56.6948473Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:37:56.6949384Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:37:56.6949761Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:37:56.6950231Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:37:56.9116739Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:37:56.9162433Z   test_freezing_weights_with_nested_trunk_True_freezing_method_FreezingMethod_GradToNone_freeze_after_wrap_fsdp_True (__main__.TestFreezingWeights) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15123
2022-05-18T03:37:56.9189299Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15124
2022-05-18T03:37:56.9212187Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15125
2022-05-18T03:37:56.9235876Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 15126
2022-05-18T03:37:57.5111750Z dist init r=2, world=4
2022-05-18T03:37:57.5505547Z dist init r=0, world=4
2022-05-18T03:37:57.5546461Z dist init r=3, world=4
2022-05-18T03:37:57.5577158Z dist init r=1, world=4
2022-05-18T03:37:57.5756274Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:37:57.5887558Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:37:57.5888249Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:37:57.5888808Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:37:57.5889777Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:57.5890477Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:57.5891066Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:57.5959794Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:57.5995968Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:37:57.5996382Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:37:57.5996967Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:37:57.5997315Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:37:57.8262423Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:37:57.8306179Z   test_freezing_weights_with_nested_trunk_True_freezing_method_FreezingMethod_RequiresGrad_freeze_after_wrap_fsdp_False (__main__.TestFreezingWeights) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15179
2022-05-18T03:37:57.8332590Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15180
2022-05-18T03:37:57.8355788Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15181
2022-05-18T03:37:57.8378818Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 15182
2022-05-18T03:37:58.4429944Z dist init r=2, world=4
2022-05-18T03:37:58.4487283Z dist init r=0, world=4
2022-05-18T03:37:58.4583489Z dist init r=3, world=4
2022-05-18T03:37:58.4873642Z dist init r=1, world=4
2022-05-18T03:37:58.5200208Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:37:58.5300724Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:37:58.5402898Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:37:58.5404060Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:58.5404901Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:58.5405382Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:37:58.5405974Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:58.5406692Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:58.5411293Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:37:58.5411956Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:37:58.5412606Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:37:58.5413191Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:37:58.7405572Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:37:58.7457834Z   test_freezing_weights_with_nested_trunk_True_freezing_method_FreezingMethod_RequiresGrad_freeze_after_wrap_fsdp_True (__main__.TestFreezingWeights) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15235
2022-05-18T03:37:58.7478654Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15236
2022-05-18T03:37:58.7502399Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15237
2022-05-18T03:37:58.7526894Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 15238
2022-05-18T03:37:59.3302113Z dist init r=1, world=4
2022-05-18T03:37:59.3839600Z dist init r=3, world=4
2022-05-18T03:37:59.4081456Z dist init r=0, world=4
2022-05-18T03:37:59.4170496Z dist init r=2, world=4
2022-05-18T03:37:59.4349990Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:37:59.4451537Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:37:59.4552549Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:37:59.4553006Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:37:59.4553680Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:59.4554342Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:59.4554872Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:59.4555775Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:37:59.4660401Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:37:59.4661102Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:37:59.4661457Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:37:59.4661798Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:37:59.6553842Z skip: Need at least 2 CUDA devices (0.915s)
2022-05-18T03:37:59.6554052Z 
2022-05-18T03:37:59.6554479Z ----------------------------------------------------------------------
2022-05-18T03:37:59.6554750Z Ran 8 tests in 7.601s
2022-05-18T03:37:59.6554865Z 
2022-05-18T03:37:59.6554941Z OK (skipped=8)
2022-05-18T03:37:59.6555049Z 
2022-05-18T03:37:59.6555140Z Generating XML reports...
2022-05-18T03:37:59.6594494Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_freezing_weights/TEST-TestFreezingWeights-20220518033752.xml
2022-05-18T03:37:59.8450293Z Running distributed/fsdp/test_fsdp_grad_acc ... [2022-05-18 03:37:59.844688]
2022-05-18T03:37:59.8451212Z Executing ['/opt/conda/bin/python', 'distributed/fsdp/test_fsdp_grad_acc.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-05-18 03:37:59.844768]
2022-05-18T03:38:00.4213073Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_grad_acc
2022-05-18T03:38:00.4224798Z 
2022-05-18T03:38:00.4224927Z Running tests...
2022-05-18T03:38:00.4225548Z ----------------------------------------------------------------------
2022-05-18T03:38:00.4237099Z   test_grad_acc_configs_[(use_no_sync=False,num_iters=3),(use_no_sync=True,num_iters=3),(use_no_sync=False,num_iters=3)]_cpu_offload_CPUOffload(offload_params=False)_backward_prefetch_BackwardPrefetch_BACKWARD_POST (__main__.TestGradAcc)
2022-05-18T03:38:00.7018806Z Tests gradient accumulation. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15302
2022-05-18T03:38:00.7040403Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15303
2022-05-18T03:38:00.7062732Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15304
2022-05-18T03:38:00.7086636Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 15305
2022-05-18T03:38:01.3278982Z dist init r=3, world=4
2022-05-18T03:38:01.3703955Z dist init r=2, world=4
2022-05-18T03:38:01.3802302Z dist init r=0, world=4
2022-05-18T03:38:01.3876120Z dist init r=1, world=4
2022-05-18T03:38:01.4187149Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:38:01.4288696Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:38:01.4289516Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:38:01.4290096Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:38:01.4290797Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:38:01.4291326Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:38:01.4291905Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:38:01.4293151Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:38:01.4397752Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:38:01.4398369Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:38:01.4398914Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:38:01.4399429Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:38:01.6117081Z skip: Need at least 2 CUDA devices (1.189s)
2022-05-18T03:38:01.6127818Z   test_grad_acc_configs_[(use_no_sync=False,num_iters=3),(use_no_sync=True,num_iters=3),(use_no_sync=False,num_iters=3)]_cpu_offload_CPUOffload(offload_params=False)_backward_prefetch_BackwardPrefetch_BACKWARD_PRE (__main__.TestGradAcc)
2022-05-18T03:38:01.6163974Z Tests gradient accumulation. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15358
2022-05-18T03:38:01.6190269Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15359
2022-05-18T03:38:01.6213478Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15360
2022-05-18T03:38:01.6237603Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 15361
2022-05-18T03:38:02.2207016Z dist init r=2, world=4
2022-05-18T03:38:02.2317824Z dist init r=1, world=4
2022-05-18T03:38:02.2881787Z dist init r=3, world=4
2022-05-18T03:38:02.2882062Z dist init r=0, world=4
2022-05-18T03:38:02.3092508Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:38:02.3193859Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:38:02.3295269Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:38:02.3295829Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:38:02.3297032Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:38:02.3297937Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:38:02.3298754Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:38:02.3299415Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:38:02.3404442Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:38:02.3405022Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:38:02.3405551Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:38:02.3406088Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:38:02.5263671Z skip: Need at least 2 CUDA devices (0.915s)
2022-05-18T03:38:02.5275134Z   test_grad_acc_configs_[(use_no_sync=False,num_iters=3),(use_no_sync=True,num_iters=3),(use_no_sync=False,num_iters=3)]_cpu_offload_CPUOffload(offload_params=False)_backward_prefetch_None (__main__.TestGradAcc)
2022-05-18T03:38:02.5311179Z Tests gradient accumulation. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15414
2022-05-18T03:38:02.5336572Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15415
2022-05-18T03:38:02.5359991Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15416
2022-05-18T03:38:02.5384202Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 15417
2022-05-18T03:38:03.1045563Z dist init r=2, world=4
2022-05-18T03:38:03.1153272Z dist init r=3, world=4
2022-05-18T03:38:03.1245592Z dist init r=0, world=4
2022-05-18T03:38:03.1813745Z dist init r=1, world=4
2022-05-18T03:38:03.2124020Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:38:03.2124586Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:38:03.2225671Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:38:03.2226257Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:38:03.2227140Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:38:03.2227884Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:38:03.2228506Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:38:03.2230484Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:38:03.2334137Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:38:03.2334720Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:38:03.2335260Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:38:03.2335804Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:38:03.4411557Z skip: Need at least 2 CUDA devices (0.915s)
2022-05-18T03:38:03.4422268Z   test_grad_acc_configs_[(use_no_sync=False,num_iters=3),(use_no_sync=True,num_iters=3),(use_no_sync=False,num_iters=3)]_cpu_offload_CPUOffload(offload_params=True)_backward_prefetch_BackwardPrefetch_BACKWARD_POST (__main__.TestGradAcc)
2022-05-18T03:38:03.4458626Z Tests gradient accumulation. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15470
2022-05-18T03:38:03.4484021Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15471
2022-05-18T03:38:03.4507612Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15472
2022-05-18T03:38:03.4531841Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 15473
2022-05-18T03:38:04.0821357Z dist init r=3, world=4
2022-05-18T03:38:04.0822684Z dist init r=0, world=4
2022-05-18T03:38:04.1329533Z dist init r=2, world=4
2022-05-18T03:38:04.1432941Z dist init r=1, world=4
2022-05-18T03:38:04.1634488Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:38:04.1734155Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:38:04.1836827Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:38:04.1837451Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:38:04.1838295Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:38:04.1838912Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:38:04.1840765Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:38:04.1841292Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:38:04.1945229Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:38:04.1945902Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:38:04.1946245Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:38:04.1946740Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:38:04.3557020Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:38:04.3567625Z   test_grad_acc_configs_[(use_no_sync=False,num_iters=3),(use_no_sync=True,num_iters=3),(use_no_sync=False,num_iters=3)]_cpu_offload_CPUOffload(offload_params=True)_backward_prefetch_BackwardPrefetch_BACKWARD_PRE (__main__.TestGradAcc)
2022-05-18T03:38:04.3604608Z Tests gradient accumulation. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15526
2022-05-18T03:38:04.3630678Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15527
2022-05-18T03:38:04.3654094Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15528
2022-05-18T03:38:04.3678798Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 15529
2022-05-18T03:38:04.9386511Z dist init r=2, world=4
2022-05-18T03:38:04.9646765Z dist init r=0, world=4
2022-05-18T03:38:04.9978420Z dist init r=3, world=4
2022-05-18T03:38:05.0073104Z dist init r=1, world=4
2022-05-18T03:38:05.0459821Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:38:05.0560778Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:38:05.0561503Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:38:05.0562196Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:38:05.0563188Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:38:05.0564153Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:38:05.0565137Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:38:05.0566061Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:38:05.0669638Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:38:05.0670304Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:38:05.0670828Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:38:05.0671388Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:38:05.2705329Z skip: Need at least 2 CUDA devices (0.915s)
2022-05-18T03:38:05.2716024Z   test_grad_acc_configs_[(use_no_sync=False,num_iters=3),(use_no_sync=True,num_iters=3),(use_no_sync=False,num_iters=3)]_cpu_offload_CPUOffload(offload_params=True)_backward_prefetch_None (__main__.TestGradAcc)
2022-05-18T03:38:05.2752309Z Tests gradient accumulation. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15582
2022-05-18T03:38:05.2779347Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15583
2022-05-18T03:38:05.2802354Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15584
2022-05-18T03:38:05.2826405Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 15585
2022-05-18T03:38:05.8898405Z dist init r=3, world=4
2022-05-18T03:38:05.8946335Z dist init r=2, world=4
2022-05-18T03:38:05.9062272Z dist init r=1, world=4
2022-05-18T03:38:05.9407230Z dist init r=0, world=4
2022-05-18T03:38:05.9609658Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:38:05.9710757Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:38:05.9813621Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:38:05.9814548Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:38:05.9815568Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:38:05.9816283Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:38:05.9816822Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:38:05.9817331Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:38:05.9823538Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:38:05.9823947Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:38:05.9824490Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:38:05.9825056Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:38:06.1853149Z skip: Need at least 2 CUDA devices (0.915s)
2022-05-18T03:38:06.1864041Z   test_grad_acc_configs_[(use_no_sync=True,num_iters=3),(use_no_sync=False,num_iters=3),(use_no_sync=True,num_iters=3)]_cpu_offload_CPUOffload(offload_params=False)_backward_prefetch_BackwardPrefetch_BACKWARD_POST (__main__.TestGradAcc)
2022-05-18T03:38:06.1900256Z Tests gradient accumulation. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15638
2022-05-18T03:38:06.1926749Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15639
2022-05-18T03:38:06.1950208Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15640
2022-05-18T03:38:06.1974114Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 15641
2022-05-18T03:38:06.7695842Z dist init r=2, world=4
2022-05-18T03:38:06.7705116Z dist init r=0, world=4
2022-05-18T03:38:06.7913968Z dist init r=3, world=4
2022-05-18T03:38:06.8416611Z dist init r=1, world=4
2022-05-18T03:38:06.8627224Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:38:06.8726991Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:38:06.8727685Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:38:06.8728236Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:38:06.8729556Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:38:06.8730329Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:38:06.8731479Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:38:06.8732362Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:38:06.8835824Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:38:06.8836646Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:38:06.8837545Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:38:06.8838456Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:38:07.1000808Z skip: Need at least 2 CUDA devices (0.915s)
2022-05-18T03:38:07.1011049Z   test_grad_acc_configs_[(use_no_sync=True,num_iters=3),(use_no_sync=False,num_iters=3),(use_no_sync=True,num_iters=3)]_cpu_offload_CPUOffload(offload_params=False)_backward_prefetch_BackwardPrefetch_BACKWARD_PRE (__main__.TestGradAcc)
2022-05-18T03:38:07.1047893Z Tests gradient accumulation. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15694
2022-05-18T03:38:07.1074099Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15695
2022-05-18T03:38:07.1097432Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15696
2022-05-18T03:38:07.1121709Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 15697
2022-05-18T03:38:07.6982855Z dist init r=1, world=4
2022-05-18T03:38:07.7002267Z dist init r=0, world=4
2022-05-18T03:38:07.7096270Z dist init r=3, world=4
2022-05-18T03:38:07.7557539Z dist init r=2, world=4
2022-05-18T03:38:07.7708127Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:38:07.7796040Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:38:07.7796725Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:38:07.7797364Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:38:07.7798263Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:38:07.7800064Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:38:07.7800584Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:38:07.7811747Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:38:07.7907648Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:38:07.7908273Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:38:07.7908837Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:38:07.7909187Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:38:08.0148319Z skip: Need at least 2 CUDA devices (0.915s)
2022-05-18T03:38:08.0159162Z   test_grad_acc_configs_[(use_no_sync=True,num_iters=3),(use_no_sync=False,num_iters=3),(use_no_sync=True,num_iters=3)]_cpu_offload_CPUOffload(offload_params=False)_backward_prefetch_None (__main__.TestGradAcc)
2022-05-18T03:38:08.0195422Z Tests gradient accumulation. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15750
2022-05-18T03:38:08.0224002Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15751
2022-05-18T03:38:08.0247146Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15752
2022-05-18T03:38:08.0271313Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 15753
2022-05-18T03:38:08.6103303Z dist init r=1, world=4
2022-05-18T03:38:08.6144198Z dist init r=3, world=4
2022-05-18T03:38:08.6496130Z dist init r=2, world=4
2022-05-18T03:38:08.6714266Z dist init r=0, world=4
2022-05-18T03:38:08.7015110Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:38:08.7219612Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:38:08.7321106Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:38:08.7322105Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:38:08.7322801Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:38:08.7325339Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:38:08.7326089Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:38:08.7326686Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:38:08.7331564Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:38:08.7332094Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:38:08.7332616Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:38:08.7333098Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:38:08.9298354Z skip: Need at least 2 CUDA devices (0.915s)
2022-05-18T03:38:08.9309182Z   test_grad_acc_configs_[(use_no_sync=True,num_iters=3),(use_no_sync=False,num_iters=3),(use_no_sync=True,num_iters=3)]_cpu_offload_CPUOffload(offload_params=True)_backward_prefetch_BackwardPrefetch_BACKWARD_POST (__main__.TestGradAcc)
2022-05-18T03:38:08.9345854Z Tests gradient accumulation. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15806
2022-05-18T03:38:08.9372837Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15807
2022-05-18T03:38:08.9396823Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15808
2022-05-18T03:38:08.9421373Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 15809
2022-05-18T03:38:09.5548345Z dist init r=2, world=4
2022-05-18T03:38:09.5548723Z dist init r=1, world=4
2022-05-18T03:38:09.5629269Z dist init r=0, world=4
2022-05-18T03:38:09.5750335Z dist init r=3, world=4
2022-05-18T03:38:09.5858714Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:38:09.5958948Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:38:09.6060908Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:38:09.6061488Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:38:09.6063711Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:38:09.6064539Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:38:09.6065046Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:38:09.6065559Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:38:09.6169606Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:38:09.6170379Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:38:09.6170931Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:38:09.6171382Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:38:09.8447808Z skip: Need at least 2 CUDA devices (0.915s)
2022-05-18T03:38:09.8459274Z   test_grad_acc_configs_[(use_no_sync=True,num_iters=3),(use_no_sync=False,num_iters=3),(use_no_sync=True,num_iters=3)]_cpu_offload_CPUOffload(offload_params=True)_backward_prefetch_BackwardPrefetch_BACKWARD_PRE (__main__.TestGradAcc)
2022-05-18T03:38:09.8496025Z Tests gradient accumulation. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15862
2022-05-18T03:38:09.8522281Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15863
2022-05-18T03:38:09.8545685Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15864
2022-05-18T03:38:09.8569732Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 15865
2022-05-18T03:38:10.4649788Z dist init r=2, world=4
2022-05-18T03:38:10.4720089Z dist init r=1, world=4
2022-05-18T03:38:10.4911144Z dist init r=0, world=4
2022-05-18T03:38:10.4921298Z dist init r=3, world=4
2022-05-18T03:38:10.5161124Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:38:10.5261642Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:38:10.5363996Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:38:10.5364548Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:38:10.5365515Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:38:10.5366322Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:38:10.5367083Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:38:10.5367637Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:38:10.5373827Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:38:10.5374498Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:38:10.5374917Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:38:10.5376237Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:38:10.7596499Z skip: Need at least 2 CUDA devices (0.915s)
2022-05-18T03:38:10.7607664Z   test_grad_acc_configs_[(use_no_sync=True,num_iters=3),(use_no_sync=False,num_iters=3),(use_no_sync=True,num_iters=3)]_cpu_offload_CPUOffload(offload_params=True)_backward_prefetch_None (__main__.TestGradAcc)
2022-05-18T03:38:10.7646320Z Tests gradient accumulation. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15918
2022-05-18T03:38:10.7673317Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15919
2022-05-18T03:38:10.7696960Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15920
2022-05-18T03:38:10.7721950Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 15921
2022-05-18T03:38:11.3470235Z dist init r=3, world=4
2022-05-18T03:38:11.3534236Z dist init r=1, world=4
2022-05-18T03:38:11.3955051Z dist init r=2, world=4
2022-05-18T03:38:11.4000120Z dist init r=0, world=4
2022-05-18T03:38:11.4181950Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:38:11.4283518Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:38:11.4384481Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:38:11.4385147Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:38:11.4387324Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:38:11.4388155Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:38:11.4388770Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:38:11.4389441Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:38:11.4394336Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:38:11.4394914Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:38:11.4395486Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:38:11.4396033Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:38:11.6747998Z skip: Need at least 2 CUDA devices (0.915s)
2022-05-18T03:38:11.6748377Z 
2022-05-18T03:38:11.6748890Z ----------------------------------------------------------------------
2022-05-18T03:38:11.6749153Z Ran 12 tests in 11.252s
2022-05-18T03:38:11.6749270Z 
2022-05-18T03:38:11.6749346Z OK (skipped=12)
2022-05-18T03:38:11.6749442Z 
2022-05-18T03:38:11.6749530Z Generating XML reports...
2022-05-18T03:38:11.6799385Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_grad_acc/TEST-TestGradAcc-20220518033800.xml
2022-05-18T03:38:11.8742659Z Running distributed/fsdp/test_fsdp_ignored_modules ... [2022-05-18 03:38:11.873865]
2022-05-18T03:38:11.8743454Z Executing ['/opt/conda/bin/python', 'distributed/fsdp/test_fsdp_ignored_modules.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-05-18 03:38:11.873948]
2022-05-18T03:38:12.4506815Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_ignored_modules
2022-05-18T03:38:12.4522776Z 
2022-05-18T03:38:12.4523241Z Running tests...
2022-05-18T03:38:12.4523624Z ----------------------------------------------------------------------
2022-05-18T03:38:12.4530029Z   test_ignored_modules_invalid (__main__.TestFSDPIgnoredModules)
2022-05-18T03:38:12.7320680Z Tests that passing an FSDP module as an ignored module or the ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15985
2022-05-18T03:38:12.7343173Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15986
2022-05-18T03:38:12.7365561Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15987
2022-05-18T03:38:12.7390188Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 15988
2022-05-18T03:38:13.3698328Z dist init r=1, world=4
2022-05-18T03:38:13.3917066Z dist init r=3, world=4
2022-05-18T03:38:13.3986147Z dist init r=0, world=4
2022-05-18T03:38:13.4071137Z dist init r=2, world=4
2022-05-18T03:38:13.4225662Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:38:13.4326839Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:38:13.4411221Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:38:13.4411669Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:38:13.4412499Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:38:13.4413120Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:38:13.4429849Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:38:13.4430613Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:38:13.4518644Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:38:13.4519327Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:38:13.4519950Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:38:13.4522208Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:38:13.6422169Z skip: Need at least 2 CUDA devices (1.190s)
2022-05-18T03:38:13.6433933Z   test_ignored_modules_nested (__main__.TestFSDPIgnoredModules)
2022-05-18T03:38:13.6471596Z Tests that passing a module with nested FSDP modules does not ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16041
2022-05-18T03:38:13.6498254Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16042
2022-05-18T03:38:13.6521931Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16043
2022-05-18T03:38:13.6546034Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 16044
2022-05-18T03:38:14.2552213Z dist init r=1, world=4
2022-05-18T03:38:14.2562217Z dist init r=0, world=4
2022-05-18T03:38:14.2668162Z dist init r=3, world=4
2022-05-18T03:38:14.2981225Z dist init r=2, world=4
2022-05-18T03:38:14.3178492Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:38:14.3274475Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:38:14.3376425Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:38:14.3376915Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:38:14.3377534Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:38:14.3378057Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:38:14.3378584Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:38:14.3381622Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:38:14.3483433Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:38:14.3484109Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:38:14.3484708Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:38:14.3485061Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:38:14.5573288Z skip: Need at least 2 CUDA devices (0.915s)
2022-05-18T03:38:14.5584500Z   test_ignored_modules_transformer (__main__.TestFSDPIgnoredModules)
2022-05-18T03:38:14.5620248Z Tests that ignored modules' parameters are not flattened for a ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16097
2022-05-18T03:38:14.5645991Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16098
2022-05-18T03:38:14.5669672Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16099
2022-05-18T03:38:14.5694671Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 16100
2022-05-18T03:38:15.1596211Z dist init r=1, world=4
2022-05-18T03:38:15.2115338Z dist init r=3, world=4
2022-05-18T03:38:15.2121596Z dist init r=0, world=4
2022-05-18T03:38:15.2136498Z dist init r=2, world=4
2022-05-18T03:38:15.2323539Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:38:15.2526329Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:38:15.2527452Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:38:15.2528082Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:38:15.2528868Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:38:15.2529506Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:38:15.2530194Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:38:15.2530921Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:38:15.2535470Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:38:15.2536047Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:38:15.2536729Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:38:15.2540129Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:38:15.4721886Z skip: Need at least 2 CUDA devices (0.915s)
2022-05-18T03:38:15.4722198Z 
2022-05-18T03:38:15.4722703Z ----------------------------------------------------------------------
2022-05-18T03:38:15.4722968Z Ran 3 tests in 3.020s
2022-05-18T03:38:15.4723087Z 
2022-05-18T03:38:15.4723169Z OK (skipped=3)
2022-05-18T03:38:15.4723278Z 
2022-05-18T03:38:15.4723361Z Generating XML reports...
2022-05-18T03:38:15.4761142Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_ignored_modules/TEST-TestFSDPIgnoredModules-20220518033812.xml
2022-05-18T03:38:15.6691270Z Running distributed/fsdp/test_fsdp_input ... [2022-05-18 03:38:15.668713]
2022-05-18T03:38:15.6692171Z Executing ['/opt/conda/bin/python', 'distributed/fsdp/test_fsdp_input.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-05-18 03:38:15.668798]
2022-05-18T03:38:16.2390037Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_input
2022-05-18T03:38:16.2405833Z 
2022-05-18T03:38:16.2406302Z Running tests...
2022-05-18T03:38:16.2406727Z ----------------------------------------------------------------------
2022-05-18T03:38:16.2418768Z   test_input_type_dict (__main__.TestInput)
2022-05-18T03:38:16.5215827Z Test FSDP with input being a list or a dict, only single GPU. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16164
2022-05-18T03:38:17.0883052Z dist init r=0, world=1
2022-05-18T03:38:17.0890885Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:38:17.0891839Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-05-18T03:38:17.0894924Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:38:17.2236230Z skip: Need at least 1 CUDA device (0.983s)
2022-05-18T03:38:17.2249198Z   test_input_type_list (__main__.TestInput)
2022-05-18T03:38:17.2286323Z Test FSDP with input being a list or a dict, only single GPU. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16178
2022-05-18T03:38:17.7979605Z dist init r=0, world=1
2022-05-18T03:38:17.7988052Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:38:17.7988762Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-05-18T03:38:17.7992158Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:38:17.9304736Z skip: Need at least 1 CUDA device (0.707s)
2022-05-18T03:38:17.9305001Z 
2022-05-18T03:38:17.9305463Z ----------------------------------------------------------------------
2022-05-18T03:38:17.9305865Z Ran 2 tests in 1.690s
2022-05-18T03:38:17.9306038Z 
2022-05-18T03:38:17.9306777Z OK (skipped=2)
2022-05-18T03:38:17.9306966Z 
2022-05-18T03:38:17.9307117Z Generating XML reports...
2022-05-18T03:38:17.9343219Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_input/TEST-TestInput-20220518033816.xml
2022-05-18T03:38:18.1241273Z Running distributed/fsdp/test_fsdp_memory ... [2022-05-18 03:38:18.123697]
2022-05-18T03:38:18.1242245Z Executing ['/opt/conda/bin/python', 'distributed/fsdp/test_fsdp_memory.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-05-18 03:38:18.123783]
2022-05-18T03:38:18.7009041Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_memory
2022-05-18T03:38:18.7025202Z 
2022-05-18T03:38:18.7025332Z Running tests...
2022-05-18T03:38:18.7025794Z ----------------------------------------------------------------------
2022-05-18T03:38:18.9806936Z   test_fsdp_memory_ckpt_ckpt (__main__.TestFSDPMemory) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16203
2022-05-18T03:38:18.9828416Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16204
2022-05-18T03:38:19.5666072Z dist init r=1, world=2
2022-05-18T03:38:19.6037780Z dist init r=0, world=2
2022-05-18T03:38:19.6176419Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:38:19.6177027Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:38:19.6177729Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:38:19.6178261Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:38:19.6182448Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:38:19.6182821Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:38:19.7852276Z skip: Need at least 2 CUDA devices (1.082s)
2022-05-18T03:38:19.7903067Z   test_fsdp_memory_ckpt_no_ckpt (__main__.TestFSDPMemory) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16231
2022-05-18T03:38:19.7928080Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16232
2022-05-18T03:38:20.3725068Z dist init r=0, world=2
2022-05-18T03:38:20.3738195Z dist init r=1, world=2
2022-05-18T03:38:20.3846303Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:38:20.3846714Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:38:20.3847354Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:38:20.3847886Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:38:20.3952494Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:38:20.3952955Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:38:20.5950399Z skip: Need at least 2 CUDA devices (0.810s)
2022-05-18T03:38:20.5950781Z 
2022-05-18T03:38:20.5951504Z ----------------------------------------------------------------------
2022-05-18T03:38:20.5951767Z Ran 2 tests in 1.892s
2022-05-18T03:38:20.5951887Z 
2022-05-18T03:38:20.5951959Z OK (skipped=2)
2022-05-18T03:38:20.5952055Z 
2022-05-18T03:38:20.5952140Z Generating XML reports...
2022-05-18T03:38:20.5990143Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_memory/TEST-TestFSDPMemory-20220518033818.xml
2022-05-18T03:38:20.7951170Z Running distributed/fsdp/test_fsdp_meta ... [2022-05-18 03:38:20.794705]
2022-05-18T03:38:20.7952095Z Executing ['/opt/conda/bin/python', 'distributed/fsdp/test_fsdp_meta.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-05-18 03:38:20.794798]
2022-05-18T03:38:21.3684851Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_meta
2022-05-18T03:38:21.3697087Z 
2022-05-18T03:38:21.3697547Z Running tests...
2022-05-18T03:38:21.3697956Z ----------------------------------------------------------------------
2022-05-18T03:38:21.6481575Z   test_bad_arg_meta (__main__.TestFSDPWithMetaDevice) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16270
2022-05-18T03:38:21.6503060Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16271
2022-05-18T03:38:22.2515881Z dist init r=0, world=2
2022-05-18T03:38:22.2539640Z dist init r=1, world=2
2022-05-18T03:38:22.2647867Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:38:22.2648278Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:38:22.2648912Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:38:22.2649429Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:38:22.2755240Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:38:22.2755832Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:38:22.4527371Z skip: Need at least 2 CUDA devices (1.083s)
2022-05-18T03:38:22.4532963Z   test_bad_arg_torchdistx (__main__.TestFSDPWithMetaDevice) ... skip: Test requires torchdistX: https://github.com/pytorch/torchdistX (0.001s)
2022-05-18T03:38:22.4571254Z   test_nested_model_with_meta_device_default_init_auto_wrap_False (__main__.TestFSDPWithMetaDevice) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16298
2022-05-18T03:38:22.4596894Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16299
2022-05-18T03:38:23.0329417Z dist init r=0, world=2
2022-05-18T03:38:23.0655438Z dist init r=1, world=2
2022-05-18T03:38:23.0763514Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:38:23.0764010Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:38:23.0764801Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:38:23.0765339Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:38:23.0869778Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:38:23.0870138Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:38:23.2617702Z skip: Need at least 2 CUDA devices (0.808s)
2022-05-18T03:38:23.2658455Z   test_nested_model_with_meta_device_default_init_auto_wrap_True (__main__.TestFSDPWithMetaDevice) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16326
2022-05-18T03:38:23.2684668Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16327
2022-05-18T03:38:23.8390816Z dist init r=0, world=2
2022-05-18T03:38:23.8409319Z dist init r=1, world=2
2022-05-18T03:38:23.8599100Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:38:23.8599615Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:38:23.8600238Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:38:23.8600759Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:38:23.8705108Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:38:23.8705649Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:38:24.0706946Z skip: Need at least 2 CUDA devices (0.809s)
2022-05-18T03:38:24.0747608Z   test_nested_model_with_meta_device_reset_params_auto_wrap_False (__main__.TestFSDPWithMetaDevice) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16354
2022-05-18T03:38:24.0773142Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16355
2022-05-18T03:38:24.6583237Z dist init r=0, world=2
2022-05-18T03:38:24.6939726Z dist init r=1, world=2
2022-05-18T03:38:24.7048136Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:38:24.7048726Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:38:24.7049439Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:38:24.7049977Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:38:24.7154729Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:38:24.7155141Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:38:24.8794825Z skip: Need at least 2 CUDA devices (0.809s)
2022-05-18T03:38:24.8836582Z   test_nested_model_with_meta_device_reset_params_auto_wrap_True (__main__.TestFSDPWithMetaDevice) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16382
2022-05-18T03:38:24.8862843Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16383
2022-05-18T03:38:25.4728281Z dist init r=0, world=2
2022-05-18T03:38:25.5059611Z dist init r=1, world=2
2022-05-18T03:38:25.5238995Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:38:25.5239486Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:38:25.5240136Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:38:25.5240674Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:38:25.5344778Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:38:25.5345255Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:38:25.6883823Z skip: Need at least 2 CUDA devices (0.809s)
2022-05-18T03:38:25.6890568Z   test_nested_model_with_torchdistX_default_init_auto_wrap_False (__main__.TestFSDPWithMetaDevice) ... skip: Test requires torchdistX: https://github.com/pytorch/torchdistX (0.001s)
2022-05-18T03:38:25.6894496Z   test_nested_model_with_torchdistX_default_init_auto_wrap_True (__main__.TestFSDPWithMetaDevice) ... skip: Test requires torchdistX: https://github.com/pytorch/torchdistX (0.000s)
2022-05-18T03:38:25.6899209Z   test_nested_model_with_torchdistX_init_fn_auto_wrap_False (__main__.TestFSDPWithMetaDevice) ... skip: Test requires torchdistX: https://github.com/pytorch/torchdistX (0.000s)
2022-05-18T03:38:25.6903421Z   test_nested_model_with_torchdistX_init_fn_auto_wrap_True (__main__.TestFSDPWithMetaDevice) ... skip: Test requires torchdistX: https://github.com/pytorch/torchdistX (0.000s)
2022-05-18T03:38:25.6940625Z   test_simple_model_with_meta_device_default_init (__main__.TestFSDPWithMetaDevice) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16410
2022-05-18T03:38:25.6966078Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16411
2022-05-18T03:38:26.2632825Z dist init r=1, world=2
2022-05-18T03:38:26.2664542Z dist init r=0, world=2
2022-05-18T03:38:26.2873985Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:38:26.2874419Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:38:26.2875140Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:38:26.2875663Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:38:26.2878837Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:38:26.2879379Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:38:26.4988229Z skip: Need at least 2 CUDA devices (0.808s)
2022-05-18T03:38:26.5029816Z   test_simple_model_with_meta_device_reset_params (__main__.TestFSDPWithMetaDevice) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16438
2022-05-18T03:38:26.5057757Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16439
2022-05-18T03:38:27.0919271Z dist init r=0, world=2
2022-05-18T03:38:27.1240440Z dist init r=1, world=2
2022-05-18T03:38:27.1429805Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:38:27.1430306Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:38:27.1430931Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:38:27.1431458Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:38:27.1534641Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:38:27.1535161Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:38:27.3080056Z skip: Need at least 2 CUDA devices (0.809s)
2022-05-18T03:38:27.3085008Z   test_simple_model_with_torchdistX_default_init (__main__.TestFSDPWithMetaDevice) ... skip: Test requires torchdistX: https://github.com/pytorch/torchdistX (0.001s)
2022-05-18T03:38:27.3088719Z   test_simple_model_with_torchdistX_init_fn (__main__.TestFSDPWithMetaDevice) ... skip: Test requires torchdistX: https://github.com/pytorch/torchdistX (0.000s)
2022-05-18T03:38:27.3089137Z 
2022-05-18T03:38:27.3089539Z ----------------------------------------------------------------------
2022-05-18T03:38:27.3089897Z Ran 14 tests in 5.939s
2022-05-18T03:38:27.3090108Z 
2022-05-18T03:38:27.3090250Z OK (skipped=14)
2022-05-18T03:38:27.3090450Z 
2022-05-18T03:38:27.3090585Z Generating XML reports...
2022-05-18T03:38:27.3138050Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_meta/TEST-TestFSDPWithMetaDevice-20220518033821.xml
2022-05-18T03:38:27.5012589Z Running distributed/fsdp/test_fsdp_misc ... [2022-05-18 03:38:27.500867]
2022-05-18T03:38:27.5013198Z Executing ['/opt/conda/bin/python', 'distributed/fsdp/test_fsdp_misc.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-05-18 03:38:27.500949]
2022-05-18T03:38:28.0729755Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_misc
2022-05-18T03:38:28.0746421Z 
2022-05-18T03:38:28.0746531Z Running tests...
2022-05-18T03:38:28.0747110Z ----------------------------------------------------------------------
2022-05-18T03:38:28.0754307Z   test_device_id_auto_wrap (__main__.TestFSDPMisc)
2022-05-18T03:38:28.3527676Z Test auto wrapping propagates the device id. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16477
2022-05-18T03:38:28.3549403Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16478
2022-05-18T03:38:28.9612512Z dist init r=0, world=2
2022-05-18T03:38:28.9638818Z dist init r=1, world=2
2022-05-18T03:38:28.9820930Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:38:28.9821393Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:38:28.9822270Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:38:28.9823296Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:38:28.9826418Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:38:28.9826992Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:38:29.1572900Z skip: Need at least 2 CUDA devices (1.082s)
2022-05-18T03:38:29.1582823Z   test_fsdp_cpu_init_stays_on_cpu (__main__.TestFSDPMisc)
2022-05-18T03:38:29.1618403Z Ensure that CPU model input stays on CPU ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16505
2022-05-18T03:38:29.1643999Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16506
2022-05-18T03:38:29.7324288Z dist init r=1, world=2
2022-05-18T03:38:29.7342176Z dist init r=0, world=2
2022-05-18T03:38:29.7552441Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:38:29.7553116Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:38:29.7553721Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:38:29.7554253Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:38:29.7656804Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:38:29.7657398Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:38:29.9666707Z skip: Need at least 2 CUDA devices (0.809s)
2022-05-18T03:38:29.9682673Z   test_fsdp_device_id_use_index_False (__main__.TestFSDPMisc)
2022-05-18T03:38:29.9719293Z If CPU module is passed into FSDP with device_id ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16533
2022-05-18T03:38:29.9745324Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16534
2022-05-18T03:38:30.5431246Z dist init r=1, world=2
2022-05-18T03:38:30.5436609Z dist init r=0, world=2
2022-05-18T03:38:30.5638952Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:38:30.5639446Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:38:30.5640058Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:38:30.5640587Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:38:30.5744176Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:38:30.5744785Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:38:30.7768082Z skip: Need at least 2 CUDA devices (0.810s)
2022-05-18T03:38:30.7783554Z   test_fsdp_device_id_use_index_True (__main__.TestFSDPMisc)
2022-05-18T03:38:30.7819407Z If CPU module is passed into FSDP with device_id ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16561
2022-05-18T03:38:30.7845779Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16562
2022-05-18T03:38:31.3629797Z dist init r=1, world=2
2022-05-18T03:38:31.3960019Z dist init r=0, world=2
2022-05-18T03:38:31.4169685Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:38:31.4170127Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:38:31.4170822Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:38:31.4171359Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:38:31.4174585Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:38:31.4175011Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:38:31.5866904Z skip: Need at least 2 CUDA devices (0.810s)
2022-05-18T03:38:31.5875860Z   test_fsdp_same_model_across_ranks (__main__.TestFSDPMisc)
2022-05-18T03:38:31.5913452Z FSDP broadcasts model from rank 0 to ensure it starts off with the same ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16589
2022-05-18T03:38:31.5938742Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16590
2022-05-18T03:38:32.1674866Z dist init r=0, world=2
2022-05-18T03:38:32.2110562Z dist init r=1, world=2
2022-05-18T03:38:32.2284870Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:38:32.2285579Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:38:32.2286232Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:38:32.2286768Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:38:32.2390624Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:38:32.2390996Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:38:32.3960701Z skip: Need at least 2 CUDA devices (0.809s)
2022-05-18T03:38:32.3967043Z   test_module_device_mismatches_device_id (__main__.TestFSDPMisc)
2022-05-18T03:38:32.4003520Z FSDP raises errors when module is on a GPU that does ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16617
2022-05-18T03:38:32.4029496Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16618
2022-05-18T03:38:32.9943568Z dist init r=1, world=2
2022-05-18T03:38:32.9983612Z dist init r=0, world=2
2022-05-18T03:38:33.0152175Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:38:33.0152637Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:38:33.0153298Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:38:33.0153831Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:38:33.0157057Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:38:33.0157641Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:38:33.2051441Z skip: Need at least 2 CUDA devices (0.809s)
2022-05-18T03:38:33.2057961Z   test_multi_device_not_supported (__main__.TestFSDPMisc)
2022-05-18T03:38:33.2094227Z FSDP throws appropriate error when we wrap multi-device module. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16645
2022-05-18T03:38:33.2119747Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16646
2022-05-18T03:38:33.7841828Z dist init r=1, world=2
2022-05-18T03:38:33.8081473Z dist init r=0, world=2
2022-05-18T03:38:33.8251385Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:38:33.8252063Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:38:33.8253032Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:38:33.8253603Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:38:33.8257507Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:38:33.8257898Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:38:34.0143069Z skip: Need at least 2 CUDA devices (0.809s)
2022-05-18T03:38:34.0150852Z   test_no_params (__main__.TestFSDPMisc)
2022-05-18T03:38:34.0186356Z Test that device_id and cpu init work if module has no params ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16673
2022-05-18T03:38:34.0212430Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16674
2022-05-18T03:38:34.6038696Z dist init r=0, world=2
2022-05-18T03:38:34.6396146Z dist init r=1, world=2
2022-05-18T03:38:34.6548744Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:38:34.6549264Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:38:34.6549875Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:38:34.6550414Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:38:34.6654114Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:38:34.6654488Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:38:34.8234804Z skip: Need at least 2 CUDA devices (0.809s)
2022-05-18T03:38:34.8234980Z 
2022-05-18T03:38:34.8235287Z ----------------------------------------------------------------------
2022-05-18T03:38:34.8235570Z Ran 8 tests in 6.749s
2022-05-18T03:38:34.8235689Z 
2022-05-18T03:38:34.8235762Z OK (skipped=8)
2022-05-18T03:38:34.8235858Z 
2022-05-18T03:38:34.8235954Z Generating XML reports...
2022-05-18T03:38:34.8278041Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_misc/TEST-TestFSDPMisc-20220518033828.xml
2022-05-18T03:38:35.0149980Z Running distributed/fsdp/test_fsdp_mixed_precision ... [2022-05-18 03:38:35.014605]
2022-05-18T03:38:35.0150963Z Executing ['/opt/conda/bin/python', 'distributed/fsdp/test_fsdp_mixed_precision.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-05-18 03:38:35.014687]
2022-05-18T03:38:35.8510821Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_mixed_precision
2022-05-18T03:38:35.8529778Z 
2022-05-18T03:38:35.8529911Z Running tests...
2022-05-18T03:38:35.8530538Z ----------------------------------------------------------------------
2022-05-18T03:38:35.8686320Z   test_mixed_precision_e2e_full_shard_mp_fp16_offload_false_prefetch_post_fp32_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16712
2022-05-18T03:38:35.8708476Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16713
2022-05-18T03:38:36.7171747Z dist init r=1, world=2
2022-05-18T03:38:36.7177847Z dist init r=0, world=2
2022-05-18T03:38:36.7387487Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:38:36.7388205Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:38:36.7389008Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:38:36.7389538Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:38:36.7393832Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:38:36.7394389Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:38:36.9735820Z skip: Need at least 2 CUDA devices (1.120s)
2022-05-18T03:38:36.9777563Z   test_mixed_precision_e2e_full_shard_mp_fp16_offload_false_prefetch_post_fp32_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16740
2022-05-18T03:38:36.9802974Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16741
2022-05-18T03:38:37.8406168Z dist init r=1, world=2
2022-05-18T03:38:37.8496808Z dist init r=0, world=2
2022-05-18T03:38:37.8614103Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:38:37.8614854Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:38:37.8615511Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:38:37.8616044Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:38:37.8619753Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:38:37.8620312Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:38:38.0827938Z skip: Need at least 2 CUDA devices (1.109s)
2022-05-18T03:38:38.0869870Z   test_mixed_precision_e2e_full_shard_mp_fp16_offload_false_prefetch_post_fp64_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16768
2022-05-18T03:38:38.0895437Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16769
2022-05-18T03:38:38.9284833Z dist init r=0, world=2
2022-05-18T03:38:38.9307698Z dist init r=1, world=2
2022-05-18T03:38:38.9493624Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:38:38.9494034Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:38:38.9494662Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:38:38.9495197Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:38:38.9598710Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:38:38.9599102Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:38:39.1922403Z skip: Need at least 2 CUDA devices (1.109s)
2022-05-18T03:38:39.1965358Z   test_mixed_precision_e2e_full_shard_mp_fp16_offload_false_prefetch_post_fp64_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16796
2022-05-18T03:38:39.1991163Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16797
2022-05-18T03:38:40.0574821Z dist init r=0, world=2
2022-05-18T03:38:40.0900862Z dist init r=1, world=2
2022-05-18T03:38:40.1085242Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:38:40.1085967Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:38:40.1086671Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:38:40.1087187Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:38:40.1191360Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:38:40.1191760Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:38:40.4016786Z skip: Need at least 2 CUDA devices (1.209s)
2022-05-18T03:38:40.4059589Z   test_mixed_precision_e2e_full_shard_mp_fp16_offload_false_prefetch_pre_fp32_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16824
2022-05-18T03:38:40.4086032Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16825
2022-05-18T03:38:41.2554214Z dist init r=0, world=2
2022-05-18T03:38:41.2874700Z dist init r=1, world=2
2022-05-18T03:38:41.2982410Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:38:41.2982814Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:38:41.2983593Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:38:41.2984127Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:38:41.3088836Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:38:41.3089531Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:38:41.5113387Z skip: Need at least 2 CUDA devices (1.110s)
2022-05-18T03:38:41.5154820Z   test_mixed_precision_e2e_full_shard_mp_fp16_offload_false_prefetch_pre_fp32_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16852
2022-05-18T03:38:41.5180967Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16853
2022-05-18T03:38:42.3625277Z dist init r=1, world=2
2022-05-18T03:38:42.3895147Z dist init r=0, world=2
2022-05-18T03:38:42.4033934Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:38:42.4034608Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:38:42.4035419Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:38:42.4035949Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:38:42.4039953Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:38:42.4040325Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:38:42.6205914Z skip: Need at least 2 CUDA devices (1.109s)
2022-05-18T03:38:42.6247137Z   test_mixed_precision_e2e_full_shard_mp_fp16_offload_false_prefetch_pre_fp64_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16880
2022-05-18T03:38:42.6272386Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16881
2022-05-18T03:38:43.4876177Z dist init r=1, world=2
2022-05-18T03:38:43.4943543Z dist init r=0, world=2
2022-05-18T03:38:43.5084762Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:38:43.5085183Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:38:43.5085900Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:38:43.5086417Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:38:43.5189895Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:38:43.5190280Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:38:43.7297144Z skip: Need at least 2 CUDA devices (1.109s)
2022-05-18T03:38:43.7341182Z   test_mixed_precision_e2e_full_shard_mp_fp16_offload_false_prefetch_pre_fp64_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16908
2022-05-18T03:38:43.7366986Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16909
2022-05-18T03:38:44.5871688Z dist init r=1, world=2
2022-05-18T03:38:44.6099225Z dist init r=0, world=2
2022-05-18T03:38:44.6307787Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:38:44.6308406Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:38:44.6309051Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:38:44.6309589Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:38:44.6314612Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:38:44.6314996Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:38:44.8393588Z skip: Need at least 2 CUDA devices (1.109s)
2022-05-18T03:38:44.8436615Z   test_mixed_precision_e2e_full_shard_mp_fp16_offload_true_prefetch_post_fp32_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16936
2022-05-18T03:38:44.8464065Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16937
2022-05-18T03:38:45.6948827Z dist init r=0, world=2
2022-05-18T03:38:45.7271797Z dist init r=1, world=2
2022-05-18T03:38:45.7459131Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:38:45.7459830Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:38:45.7460546Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:38:45.7461080Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:38:45.7565127Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:38:45.7565681Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:38:45.9488030Z skip: Need at least 2 CUDA devices (1.109s)
2022-05-18T03:38:45.9528543Z   test_mixed_precision_e2e_full_shard_mp_fp16_offload_true_prefetch_post_fp32_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16964
2022-05-18T03:38:45.9554030Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16965
2022-05-18T03:38:46.8050519Z dist init r=0, world=2
2022-05-18T03:38:46.8050900Z dist init r=1, world=2
2022-05-18T03:38:46.8261373Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:38:46.8261874Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:38:46.8262486Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:38:46.8263184Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:38:46.8366689Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:38:46.8367235Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:38:47.0578766Z skip: Need at least 2 CUDA devices (1.109s)
2022-05-18T03:38:47.0619534Z   test_mixed_precision_e2e_full_shard_mp_fp16_offload_true_prefetch_post_fp64_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16992
2022-05-18T03:38:47.0645445Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16993
2022-05-18T03:38:47.9053737Z dist init r=1, world=2
2022-05-18T03:38:47.9062308Z dist init r=0, world=2
2022-05-18T03:38:47.9261924Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:38:47.9262407Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:38:47.9263250Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:38:47.9263792Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:38:47.9367284Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:38:47.9367681Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:38:48.1669934Z skip: Need at least 2 CUDA devices (1.109s)
2022-05-18T03:38:48.1711496Z   test_mixed_precision_e2e_full_shard_mp_fp16_offload_true_prefetch_post_fp64_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17020
2022-05-18T03:38:48.1736857Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17021
2022-05-18T03:38:49.0066494Z dist init r=1, world=2
2022-05-18T03:38:49.0375280Z dist init r=0, world=2
2022-05-18T03:38:49.0576845Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:38:49.0577514Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:38:49.0578358Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:38:49.0578896Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:38:49.0585338Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:38:49.0585900Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:38:49.2761627Z skip: Need at least 2 CUDA devices (1.109s)
2022-05-18T03:38:49.2803535Z   test_mixed_precision_e2e_full_shard_mp_fp16_offload_true_prefetch_pre_fp32_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17048
2022-05-18T03:38:49.2828970Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17049
2022-05-18T03:38:50.1347760Z dist init r=0, world=2
2022-05-18T03:38:50.1699401Z dist init r=1, world=2
2022-05-18T03:38:50.1807449Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:38:50.1808440Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:38:50.1809149Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:38:50.1809676Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:38:50.1913252Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:38:50.1913633Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:38:50.3856004Z skip: Need at least 2 CUDA devices (1.109s)
2022-05-18T03:38:50.3896547Z   test_mixed_precision_e2e_full_shard_mp_fp16_offload_true_prefetch_pre_fp32_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17076
2022-05-18T03:38:50.3922252Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17077
2022-05-18T03:38:51.2399585Z dist init r=0, world=2
2022-05-18T03:38:51.2399965Z dist init r=1, world=2
2022-05-18T03:38:51.2507028Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:38:51.2509037Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:38:51.2509764Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:38:51.2608838Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:38:51.2612968Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:38:51.2613646Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:38:51.4946554Z skip: Need at least 2 CUDA devices (1.109s)
2022-05-18T03:38:51.4988028Z   test_mixed_precision_e2e_full_shard_mp_fp16_offload_true_prefetch_pre_fp64_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17104
2022-05-18T03:38:51.5014668Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17105
2022-05-18T03:38:52.3480949Z dist init r=0, world=2
2022-05-18T03:38:52.3481269Z dist init r=1, world=2
2022-05-18T03:38:52.3594211Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:38:52.3594622Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:38:52.3595382Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:38:52.3595996Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:38:52.3599689Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:38:52.3600202Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:38:52.6041882Z skip: Need at least 2 CUDA devices (1.109s)
2022-05-18T03:38:52.6090067Z   test_mixed_precision_e2e_full_shard_mp_fp16_offload_true_prefetch_pre_fp64_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17132
2022-05-18T03:38:52.6114848Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17133
2022-05-18T03:38:53.4532841Z dist init r=1, world=2
2022-05-18T03:38:53.4533185Z dist init r=0, world=2
2022-05-18T03:38:53.4742465Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:38:53.4743358Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:38:53.4744096Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:38:53.4744633Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:38:53.4749172Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:38:53.4749938Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:38:53.7139190Z skip: Need at least 2 CUDA devices (1.110s)
2022-05-18T03:38:53.7179695Z   test_mixed_precision_e2e_full_shard_mp_no_mp_offload_false_prefetch_post_fp32_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17160
2022-05-18T03:38:53.7204235Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17161
2022-05-18T03:38:54.5615498Z dist init r=1, world=2
2022-05-18T03:38:54.5629113Z dist init r=0, world=2
2022-05-18T03:38:54.5823892Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:38:54.5824615Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:38:54.5825430Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:38:54.5825967Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:38:54.5928797Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:38:54.5929452Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:38:54.8229740Z skip: Need at least 2 CUDA devices (1.109s)
2022-05-18T03:38:54.8271118Z   test_mixed_precision_e2e_full_shard_mp_no_mp_offload_false_prefetch_post_fp32_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17188
2022-05-18T03:38:54.8296614Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17189
2022-05-18T03:38:55.6815866Z dist init r=0, world=2
2022-05-18T03:38:55.6847618Z dist init r=1, world=2
2022-05-18T03:38:55.7023975Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:38:55.7024418Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:38:55.7025251Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:38:55.7025869Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:38:55.7129938Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:38:55.7130523Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:38:55.9322088Z skip: Need at least 2 CUDA devices (1.109s)
2022-05-18T03:38:55.9363119Z   test_mixed_precision_e2e_full_shard_mp_no_mp_offload_false_prefetch_post_fp64_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17216
2022-05-18T03:38:55.9389716Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17217
2022-05-18T03:38:56.7778546Z dist init r=1, world=2
2022-05-18T03:38:56.7792040Z dist init r=0, world=2
2022-05-18T03:38:56.7986411Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:38:56.7987249Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:38:56.7988415Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:38:56.7988968Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:38:56.8092315Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:38:56.8092898Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:38:57.0414037Z skip: Need at least 2 CUDA devices (1.109s)
2022-05-18T03:38:57.0455636Z   test_mixed_precision_e2e_full_shard_mp_no_mp_offload_false_prefetch_post_fp64_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17244
2022-05-18T03:38:57.0481240Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17245
2022-05-18T03:38:57.8901728Z dist init r=1, world=2
2022-05-18T03:38:57.9163432Z dist init r=0, world=2
2022-05-18T03:38:57.9372230Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:38:57.9372645Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:38:57.9373266Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:38:57.9373796Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:38:57.9379410Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:38:57.9380009Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:38:58.1505843Z skip: Need at least 2 CUDA devices (1.109s)
2022-05-18T03:38:58.1547482Z   test_mixed_precision_e2e_full_shard_mp_no_mp_offload_false_prefetch_pre_fp32_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17272
2022-05-18T03:38:58.1572715Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17273
2022-05-18T03:38:59.0156886Z dist init r=1, world=2
2022-05-18T03:38:59.0220508Z dist init r=0, world=2
2022-05-18T03:38:59.0364677Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:38:59.0365434Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:38:59.0366065Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:38:59.0366594Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:38:59.0370797Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:38:59.0371948Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:38:59.2599543Z skip: Need at least 2 CUDA devices (1.109s)
2022-05-18T03:38:59.2639985Z   test_mixed_precision_e2e_full_shard_mp_no_mp_offload_false_prefetch_pre_fp32_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17300
2022-05-18T03:38:59.2666031Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17301
2022-05-18T03:39:00.1057710Z dist init r=0, world=2
2022-05-18T03:39:00.1075751Z dist init r=1, world=2
2022-05-18T03:39:00.1265704Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:39:00.1266318Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:39:00.1267366Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:39:00.1267912Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:39:00.1271068Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:39:00.1271441Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:39:00.3689767Z skip: Need at least 2 CUDA devices (1.109s)
2022-05-18T03:39:00.3731594Z   test_mixed_precision_e2e_full_shard_mp_no_mp_offload_false_prefetch_pre_fp64_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17328
2022-05-18T03:39:00.3757034Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17329
2022-05-18T03:39:01.2088287Z dist init r=1, world=2
2022-05-18T03:39:01.2360013Z dist init r=0, world=2
2022-05-18T03:39:01.2497276Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:39:01.2497953Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:39:01.2498761Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:39:01.2499290Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:39:01.2503272Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:39:01.2503657Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:39:01.4782301Z skip: Need at least 2 CUDA devices (1.109s)
2022-05-18T03:39:01.4824328Z   test_mixed_precision_e2e_full_shard_mp_no_mp_offload_false_prefetch_pre_fp64_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17356
2022-05-18T03:39:01.4852120Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17357
2022-05-18T03:39:02.3251286Z dist init r=1, world=2
2022-05-18T03:39:02.3270287Z dist init r=0, world=2
2022-05-18T03:39:02.3459839Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:39:02.3460419Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:39:02.3461078Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:39:02.3461711Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:39:02.3466327Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:39:02.3466966Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:39:02.5876287Z skip: Need at least 2 CUDA devices (1.109s)
2022-05-18T03:39:02.5918586Z   test_mixed_precision_e2e_full_shard_mp_no_mp_offload_true_prefetch_post_fp32_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17384
2022-05-18T03:39:02.5944251Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17385
2022-05-18T03:39:03.4349817Z dist init r=0, world=2
2022-05-18T03:39:03.4365635Z dist init r=1, world=2
2022-05-18T03:39:03.4557194Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:39:03.4557899Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:39:03.4558704Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:39:03.4559478Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:39:03.4564225Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:39:03.4564758Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:39:03.6967790Z skip: Need at least 2 CUDA devices (1.109s)
2022-05-18T03:39:03.7008873Z   test_mixed_precision_e2e_full_shard_mp_no_mp_offload_true_prefetch_post_fp32_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17412
2022-05-18T03:39:03.7033942Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17413
2022-05-18T03:39:04.5382813Z dist init r=0, world=2
2022-05-18T03:39:04.5733091Z dist init r=1, world=2
2022-05-18T03:39:04.5892674Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:39:04.5893136Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:39:04.5893751Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:39:04.5894285Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:39:04.5997947Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:39:04.5998489Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:39:04.8058428Z skip: Need at least 2 CUDA devices (1.109s)
2022-05-18T03:39:04.8100993Z   test_mixed_precision_e2e_full_shard_mp_no_mp_offload_true_prefetch_post_fp64_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17440
2022-05-18T03:39:04.8127083Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17441
2022-05-18T03:39:05.6628905Z dist init r=0, world=2
2022-05-18T03:39:05.6638226Z dist init r=1, world=2
2022-05-18T03:39:05.6746477Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:39:05.6747085Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:39:05.6747789Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:39:05.6748303Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:39:05.6853059Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:39:05.6853646Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:39:05.9152218Z skip: Need at least 2 CUDA devices (1.109s)
2022-05-18T03:39:05.9193660Z   test_mixed_precision_e2e_full_shard_mp_no_mp_offload_true_prefetch_post_fp64_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17468
2022-05-18T03:39:05.9219585Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17469
2022-05-18T03:39:06.7585047Z dist init r=0, world=2
2022-05-18T03:39:06.7942272Z dist init r=1, world=2
2022-05-18T03:39:06.8094592Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:39:06.8095269Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:39:06.8095889Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:39:06.8096603Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:39:06.8200395Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:39:06.8200785Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:39:07.0244364Z skip: Need at least 2 CUDA devices (1.109s)
2022-05-18T03:39:07.0286250Z   test_mixed_precision_e2e_full_shard_mp_no_mp_offload_true_prefetch_pre_fp32_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17496
2022-05-18T03:39:07.0312686Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17497
2022-05-18T03:39:07.8725487Z dist init r=0, world=2
2022-05-18T03:39:07.8776050Z dist init r=1, world=2
2022-05-18T03:39:07.8884388Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:39:07.8884997Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:39:07.8885616Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:39:07.8886136Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:39:07.8990529Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:39:07.8991087Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:39:08.1337507Z skip: Need at least 2 CUDA devices (1.109s)
2022-05-18T03:39:08.1380084Z   test_mixed_precision_e2e_full_shard_mp_no_mp_offload_true_prefetch_pre_fp32_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17524
2022-05-18T03:39:08.1406611Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17525
2022-05-18T03:39:08.9899779Z dist init r=1, world=2
2022-05-18T03:39:08.9909248Z dist init r=0, world=2
2022-05-18T03:39:09.0107497Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:39:09.0108008Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:39:09.0108622Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:39:09.0109409Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:39:09.0113021Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:39:09.0113389Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:39:09.2431965Z skip: Need at least 2 CUDA devices (1.109s)
2022-05-18T03:39:09.2473220Z   test_mixed_precision_e2e_full_shard_mp_no_mp_offload_true_prefetch_pre_fp64_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17552
2022-05-18T03:39:09.2498415Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17553
2022-05-18T03:39:10.0881530Z dist init r=1, world=2
2022-05-18T03:39:10.0897602Z dist init r=0, world=2
2022-05-18T03:39:10.1107285Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:39:10.1107784Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:39:10.1108409Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:39:10.1108944Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:39:10.1212672Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:39:10.1213035Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:39:10.3524561Z skip: Need at least 2 CUDA devices (1.109s)
2022-05-18T03:39:10.3566255Z   test_mixed_precision_e2e_full_shard_mp_no_mp_offload_true_prefetch_pre_fp64_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17580
2022-05-18T03:39:10.3592055Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17581
2022-05-18T03:39:11.2079669Z dist init r=0, world=2
2022-05-18T03:39:11.2079943Z dist init r=1, world=2
2022-05-18T03:39:11.2187190Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:39:11.2189431Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:39:11.2190419Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:39:11.2289475Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:39:11.2293782Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:39:11.2294189Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:39:11.4615983Z skip: Need at least 2 CUDA devices (1.109s)
2022-05-18T03:39:11.4658965Z   test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_false_prefetch_post_fp32_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17608
2022-05-18T03:39:11.4684283Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17609
2022-05-18T03:39:12.3059812Z dist init r=1, world=2
2022-05-18T03:39:12.3150332Z dist init r=0, world=2
2022-05-18T03:39:12.3268706Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:39:12.3269288Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:39:12.3270019Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:39:12.3270559Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:39:12.3274040Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:39:12.3274405Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:39:12.5711664Z skip: Need at least 2 CUDA devices (1.109s)
2022-05-18T03:39:12.5753736Z   test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_false_prefetch_post_fp32_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17636
2022-05-18T03:39:12.5787165Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17637
2022-05-18T03:39:13.4211450Z dist init r=1, world=2
2022-05-18T03:39:13.4227066Z dist init r=0, world=2
2022-05-18T03:39:13.4419749Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:39:13.4420151Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:39:13.4420967Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:39:13.4421513Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:39:13.4425642Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:39:13.4426185Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:39:13.6814564Z skip: Need at least 2 CUDA devices (1.110s)
2022-05-18T03:39:13.6856286Z   test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_false_prefetch_post_fp64_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17664
2022-05-18T03:39:13.6882174Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17665
2022-05-18T03:39:14.5343724Z dist init r=0, world=2
2022-05-18T03:39:14.5714073Z dist init r=1, world=2
2022-05-18T03:39:14.5820702Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:39:14.5821446Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:39:14.5822087Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:39:14.5824488Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:39:14.5926611Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:39:14.5927244Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:39:14.7906169Z skip: Need at least 2 CUDA devices (1.109s)
2022-05-18T03:39:14.7947110Z   test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_false_prefetch_post_fp64_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17692
2022-05-18T03:39:14.7972510Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17693
2022-05-18T03:39:15.6502750Z dist init r=0, world=2
2022-05-18T03:39:15.6541100Z dist init r=1, world=2
2022-05-18T03:39:15.6710516Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:39:15.6711062Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:39:15.6711745Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:39:15.6712277Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:39:15.6715591Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:39:15.8997119Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:39:15.8997484Z skip: Need at least 2 CUDA devices (1.109s)
2022-05-18T03:39:15.9038297Z   test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_false_prefetch_pre_fp32_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17720
2022-05-18T03:39:15.9064085Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17721
2022-05-18T03:39:16.7527564Z dist init r=0, world=2
2022-05-18T03:39:16.7847112Z dist init r=1, world=2
2022-05-18T03:39:16.7955193Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:39:16.7955713Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:39:16.7956358Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:39:16.7956892Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:39:16.8061488Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:39:16.8062287Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:39:17.0091654Z skip: Need at least 2 CUDA devices (1.109s)
2022-05-18T03:39:17.0134396Z   test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_false_prefetch_pre_fp32_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17748
2022-05-18T03:39:17.0159772Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17749
2022-05-18T03:39:17.8661269Z dist init r=1, world=2
2022-05-18T03:39:17.8716052Z dist init r=0, world=2
2022-05-18T03:39:17.8869438Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:39:17.8870153Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:39:17.8870923Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:39:17.8871448Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:39:17.8874747Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:39:17.8875246Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:39:18.1185453Z skip: Need at least 2 CUDA devices (1.109s)
2022-05-18T03:39:18.1226622Z   test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_false_prefetch_pre_fp64_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17776
2022-05-18T03:39:18.1252765Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17777
2022-05-18T03:39:18.9643785Z dist init r=0, world=2
2022-05-18T03:39:18.9917040Z dist init r=1, world=2
2022-05-18T03:39:19.0023948Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:39:19.0024641Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:39:19.0025530Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:39:19.0026063Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:39:19.0129231Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:39:19.0129666Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:39:19.2280360Z skip: Need at least 2 CUDA devices (1.109s)
2022-05-18T03:39:19.2323539Z   test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_false_prefetch_pre_fp64_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17804
2022-05-18T03:39:19.2351833Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17805
2022-05-18T03:39:20.0754352Z dist init r=1, world=2
2022-05-18T03:39:20.0755765Z dist init r=0, world=2
2022-05-18T03:39:20.0965319Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:39:20.0965809Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:39:20.0966522Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:39:20.0967044Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:39:20.1071022Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:39:20.3378766Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:39:20.3379302Z skip: Need at least 2 CUDA devices (1.110s)
2022-05-18T03:39:20.3420046Z   test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_true_prefetch_post_fp32_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17832
2022-05-18T03:39:20.3445133Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17833
2022-05-18T03:39:21.1988186Z dist init r=1, world=2
2022-05-18T03:39:21.2039446Z dist init r=0, world=2
2022-05-18T03:39:21.2196250Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:39:21.2196792Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:39:21.2197409Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:39:21.2197955Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:39:21.2303112Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:39:21.2303511Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:39:21.4469819Z skip: Need at least 2 CUDA devices (1.109s)
2022-05-18T03:39:21.4512369Z   test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_true_prefetch_post_fp32_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17860
2022-05-18T03:39:21.4543456Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17861
2022-05-18T03:39:22.3022124Z dist init r=1, world=2
2022-05-18T03:39:22.3034810Z dist init r=0, world=2
2022-05-18T03:39:22.3244945Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:39:22.3245350Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:39:22.3246040Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:39:22.3254070Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:39:22.3254596Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:39:22.3254984Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:39:22.5571348Z skip: Need at least 2 CUDA devices (1.110s)
2022-05-18T03:39:22.5613638Z   test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_true_prefetch_post_fp64_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17888
2022-05-18T03:39:22.5640564Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17889
2022-05-18T03:39:23.4027339Z dist init r=1, world=2
2022-05-18T03:39:23.4148045Z dist init r=0, world=2
2022-05-18T03:39:23.4357265Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:39:23.4357924Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:39:23.4358643Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:39:23.4359444Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:39:23.4363041Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:39:23.4363710Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:39:23.6668157Z skip: Need at least 2 CUDA devices (1.110s)
2022-05-18T03:39:23.6710925Z   test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_true_prefetch_post_fp64_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17916
2022-05-18T03:39:23.6735261Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17917
2022-05-18T03:39:24.5167569Z dist init r=1, world=2
2022-05-18T03:39:24.5171964Z dist init r=0, world=2
2022-05-18T03:39:24.5380284Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:39:24.5380944Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:39:24.5381979Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:39:24.5382574Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:39:24.5386239Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:39:24.5386633Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:39:24.7762786Z skip: Need at least 2 CUDA devices (1.109s)
2022-05-18T03:39:24.7804402Z   test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_true_prefetch_pre_fp32_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17944
2022-05-18T03:39:24.7830211Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17945
2022-05-18T03:39:25.6272458Z dist init r=1, world=2
2022-05-18T03:39:25.6272906Z dist init r=0, world=2
2022-05-18T03:39:25.6481207Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:39:25.6481925Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:39:25.6482561Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:39:25.6483091Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:39:25.6487345Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:39:25.6487927Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:39:25.8857475Z skip: Need at least 2 CUDA devices (1.109s)
2022-05-18T03:39:25.8898783Z   test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_true_prefetch_pre_fp32_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17972
2022-05-18T03:39:25.8923774Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17973
2022-05-18T03:39:26.7298170Z dist init r=0, world=2
2022-05-18T03:39:26.7307165Z dist init r=1, world=2
2022-05-18T03:39:26.7415189Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:39:26.7417146Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:39:26.7418169Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:39:26.7419029Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:39:26.7420481Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:39:26.7421054Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:39:26.9951309Z skip: Need at least 2 CUDA devices (1.109s)
2022-05-18T03:39:26.9993213Z   test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_true_prefetch_pre_fp64_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18000
2022-05-18T03:39:27.0019159Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18001
2022-05-18T03:39:27.8477479Z dist init r=0, world=2
2022-05-18T03:39:27.8477818Z dist init r=1, world=2
2022-05-18T03:39:27.8584812Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:39:27.8586318Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:39:27.8587102Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:39:27.8686923Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:39:27.8691758Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:39:27.8692323Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:39:28.1045787Z skip: Need at least 2 CUDA devices (1.109s)
2022-05-18T03:39:28.1096726Z   test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_true_prefetch_pre_fp64_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18028
2022-05-18T03:39:28.1122046Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18029
2022-05-18T03:39:28.9655640Z dist init r=1, world=2
2022-05-18T03:39:28.9996069Z dist init r=0, world=2
2022-05-18T03:39:29.0204797Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:39:29.0205436Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:39:29.0206305Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:39:29.0207129Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:39:29.0209918Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:39:29.0210395Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:39:29.3149035Z skip: Need at least 2 CUDA devices (1.210s)
2022-05-18T03:39:29.3190739Z   test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_false_prefetch_post_fp32_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18056
2022-05-18T03:39:29.3216294Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18057
2022-05-18T03:39:30.1651708Z dist init r=0, world=2
2022-05-18T03:39:30.1651954Z dist init r=1, world=2
2022-05-18T03:39:30.1861170Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:39:30.1861791Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:39:30.1862421Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:39:30.1863110Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:39:30.1866141Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:39:30.1866540Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:39:30.4241592Z skip: Need at least 2 CUDA devices (1.109s)
2022-05-18T03:39:30.4282993Z   test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_false_prefetch_post_fp32_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18084
2022-05-18T03:39:30.4309055Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18085
2022-05-18T03:39:31.2684566Z dist init r=0, world=2
2022-05-18T03:39:31.2703825Z dist init r=1, world=2
2022-05-18T03:39:31.2892943Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:39:31.2893489Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:39:31.2894120Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:39:31.2894653Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:39:31.2898627Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:39:31.2899793Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:39:31.5333633Z skip: Need at least 2 CUDA devices (1.109s)
2022-05-18T03:39:31.5375411Z   test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_false_prefetch_post_fp64_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18112
2022-05-18T03:39:31.5400666Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18113
2022-05-18T03:39:32.3855639Z dist init r=0, world=2
2022-05-18T03:39:32.3855950Z dist init r=1, world=2
2022-05-18T03:39:32.3964388Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:39:32.3964996Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:39:32.3965638Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:39:32.3966157Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:39:32.4069929Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:39:32.4070475Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:39:32.6425905Z skip: Need at least 2 CUDA devices (1.109s)
2022-05-18T03:39:32.6466554Z   test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_false_prefetch_post_fp64_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18140
2022-05-18T03:39:32.6491334Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18141
2022-05-18T03:39:33.4901705Z dist init r=1, world=2
2022-05-18T03:39:33.4912127Z dist init r=0, world=2
2022-05-18T03:39:33.5121806Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:39:33.5122327Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:39:33.5123024Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:39:33.5123603Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:39:33.5126928Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:39:33.5127402Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:39:33.7516634Z skip: Need at least 2 CUDA devices (1.109s)
2022-05-18T03:39:33.7558935Z   test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_false_prefetch_pre_fp32_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18168
2022-05-18T03:39:33.7584880Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18169
2022-05-18T03:39:34.6049059Z dist init r=0, world=2
2022-05-18T03:39:34.6049271Z dist init r=1, world=2
2022-05-18T03:39:34.6257896Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:39:34.6258476Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:39:34.6259368Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:39:34.6260094Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:39:34.6263532Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:39:34.6263978Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:39:34.8609702Z skip: Need at least 2 CUDA devices (1.109s)
2022-05-18T03:39:34.8650839Z   test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_false_prefetch_pre_fp32_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18196
2022-05-18T03:39:34.8677246Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18197
2022-05-18T03:39:35.7242803Z dist init r=1, world=2
2022-05-18T03:39:35.7246044Z dist init r=0, world=2
2022-05-18T03:39:35.7457544Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:39:35.7457962Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:39:35.7458600Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:39:35.7459122Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:39:35.7563065Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:39:35.7564005Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:39:35.9701194Z skip: Need at least 2 CUDA devices (1.109s)
2022-05-18T03:39:35.9743178Z   test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_false_prefetch_pre_fp64_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18224
2022-05-18T03:39:35.9768309Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18225
2022-05-18T03:39:36.8230571Z dist init r=1, world=2
2022-05-18T03:39:36.8491117Z dist init r=0, world=2
2022-05-18T03:39:36.8640002Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:39:36.8640854Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:39:36.8642033Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:39:36.8642867Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:39:36.8647819Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:39:36.8648380Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:39:37.0793802Z skip: Need at least 2 CUDA devices (1.109s)
2022-05-18T03:39:37.0833845Z   test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_false_prefetch_pre_fp64_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18252
2022-05-18T03:39:37.0858679Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18253
2022-05-18T03:39:37.9385927Z dist init r=1, world=2
2022-05-18T03:39:37.9562902Z dist init r=0, world=2
2022-05-18T03:39:37.9771747Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:39:37.9772166Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:39:37.9772790Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:39:37.9773330Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:39:37.9777861Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:39:37.9778455Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:39:38.1883098Z skip: Need at least 2 CUDA devices (1.109s)
2022-05-18T03:39:38.1924638Z   test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_true_prefetch_post_fp32_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18280
2022-05-18T03:39:38.1950058Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18281
2022-05-18T03:39:39.0423364Z dist init r=0, world=2
2022-05-18T03:39:39.0660711Z dist init r=1, world=2
2022-05-18T03:39:39.0832480Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:39:39.0833202Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:39:39.0833880Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:39:39.0834404Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:39:39.0938167Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:39:39.0938752Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:39:39.2976643Z skip: Need at least 2 CUDA devices (1.109s)
2022-05-18T03:39:39.3018897Z   test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_true_prefetch_post_fp32_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18308
2022-05-18T03:39:39.3044894Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18309
2022-05-18T03:39:40.1486270Z dist init r=1, world=2
2022-05-18T03:39:40.1892034Z dist init r=0, world=2
2022-05-18T03:39:40.2096327Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:39:40.2096804Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:39:40.2097472Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:39:40.2098063Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:39:40.2102584Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:39:40.2103683Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:39:40.5072056Z skip: Need at least 2 CUDA devices (1.209s)
2022-05-18T03:39:40.5114247Z   test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_true_prefetch_post_fp64_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18336
2022-05-18T03:39:40.5141079Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18337
2022-05-18T03:39:41.3627609Z dist init r=0, world=2
2022-05-18T03:39:41.3941735Z dist init r=1, world=2
2022-05-18T03:39:41.4049684Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:39:41.4050138Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:39:41.4050899Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:39:41.4051426Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:39:41.4155274Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:39:41.4155657Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:39:41.6164057Z skip: Need at least 2 CUDA devices (1.109s)
2022-05-18T03:39:41.6204718Z   test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_true_prefetch_post_fp64_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18364
2022-05-18T03:39:41.6229731Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18365
2022-05-18T03:39:42.4712928Z dist init r=1, world=2
2022-05-18T03:39:42.4713293Z dist init r=0, world=2
2022-05-18T03:39:42.4820537Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:39:42.4822509Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:39:42.4823664Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:39:42.4922560Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:39:42.4927357Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:39:42.4927858Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:39:42.7258141Z skip: Need at least 2 CUDA devices (1.109s)
2022-05-18T03:39:42.7299523Z   test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_true_prefetch_pre_fp32_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18392
2022-05-18T03:39:42.7325589Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18393
2022-05-18T03:39:43.5837636Z dist init r=1, world=2
2022-05-18T03:39:43.5837863Z dist init r=0, world=2
2022-05-18T03:39:43.5944263Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:39:43.5946739Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:39:43.5947881Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:39:43.6046529Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:39:43.6051472Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:39:43.6052087Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:39:43.8351060Z skip: Need at least 2 CUDA devices (1.109s)
2022-05-18T03:39:43.8393075Z   test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_true_prefetch_pre_fp32_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18420
2022-05-18T03:39:43.8419541Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18421
2022-05-18T03:39:44.6912829Z dist init r=0, world=2
2022-05-18T03:39:44.6944485Z dist init r=1, world=2
2022-05-18T03:39:44.7120453Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:39:44.7121158Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:39:44.7121942Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:39:44.7122788Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:39:44.7126651Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:39:44.7127286Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:39:44.9445875Z skip: Need at least 2 CUDA devices (1.109s)
2022-05-18T03:39:44.9486911Z   test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_true_prefetch_pre_fp64_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18448
2022-05-18T03:39:44.9513187Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18449
2022-05-18T03:39:45.7951237Z dist init r=1, world=2
2022-05-18T03:39:45.8459137Z dist init r=0, world=2
2022-05-18T03:39:45.8668462Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:39:45.8668934Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:39:45.8669537Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:39:45.8670265Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:39:45.8675177Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:39:45.8675596Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:39:46.1541141Z skip: Need at least 2 CUDA devices (1.209s)
2022-05-18T03:39:46.1583355Z   test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_true_prefetch_pre_fp64_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18476
2022-05-18T03:39:46.1609543Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18477
2022-05-18T03:39:47.0101529Z dist init r=0, world=2
2022-05-18T03:39:47.0101911Z dist init r=1, world=2
2022-05-18T03:39:47.0309613Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:39:47.0310031Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:39:47.0310703Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:39:47.0311304Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:39:47.0414715Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:39:47.0415154Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:39:47.2635488Z skip: Need at least 2 CUDA devices (1.109s)
2022-05-18T03:39:47.2677645Z   test_mixed_precision_no_reshard_after_forward (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18504
2022-05-18T03:39:47.2704036Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18505
2022-05-18T03:39:48.1464646Z dist init r=1, world=2
2022-05-18T03:39:48.1481194Z dist init r=0, world=2
2022-05-18T03:39:48.1691281Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:39:48.1691923Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:39:48.1692523Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:39:48.1693051Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:39:48.1796454Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:39:48.1797006Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:39:48.3729403Z skip: Need at least 2 CUDA devices (1.109s)
2022-05-18T03:39:48.3742611Z   test_mixed_precision_resnet (__main__.TestFSDPMixedPrecisionSharded)
2022-05-18T03:39:48.3743355Z End to end test to ensure mixed precision + auto_wrap works ... skip: no torchvision (0.002s)
2022-05-18T03:39:48.3795961Z   test_mp_batchnorm_convert_sync_bn_False (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18532
2022-05-18T03:39:48.3821708Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18533
2022-05-18T03:39:49.2254407Z dist init r=0, world=2
2022-05-18T03:39:49.2254625Z dist init r=1, world=2
2022-05-18T03:39:49.2463446Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:39:49.2463879Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:39:49.2464542Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:39:49.2465361Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:39:49.2469001Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:39:49.2469547Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:39:49.4846943Z skip: Need at least 2 CUDA devices (1.110s)
2022-05-18T03:39:49.4900897Z   test_mp_batchnorm_convert_sync_bn_True (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18560
2022-05-18T03:39:49.4926810Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18561
2022-05-18T03:39:50.3289264Z dist init r=1, world=2
2022-05-18T03:39:50.3356821Z dist init r=0, world=2
2022-05-18T03:39:50.3496459Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:39:50.3496878Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:39:50.3497584Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:39:50.3498124Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:39:50.3501718Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:39:50.3502078Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:39:50.5951445Z skip: Need at least 2 CUDA devices (1.110s)
2022-05-18T03:39:50.5992925Z   test_mp_embedding_default (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18588
2022-05-18T03:39:50.6018193Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18589
2022-05-18T03:39:51.4595895Z dist init r=1, world=2
2022-05-18T03:39:51.4710629Z dist init r=0, world=2
2022-05-18T03:39:51.4904829Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:39:51.4905256Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:39:51.4905867Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:39:51.4906383Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:39:51.5010377Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:39:51.5010772Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:39:51.7044425Z skip: Need at least 2 CUDA devices (1.109s)
2022-05-18T03:39:51.7083966Z   test_mp_embedding_only_params_and_bufs (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18616
2022-05-18T03:39:51.7108407Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18617
2022-05-18T03:39:52.5545478Z dist init r=0, world=2
2022-05-18T03:39:52.5547917Z dist init r=1, world=2
2022-05-18T03:39:52.5754020Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:39:52.5754738Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:39:52.5755405Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:39:52.5755931Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:39:52.5759622Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:39:52.5760230Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:39:52.8134269Z skip: Need at least 2 CUDA devices (1.109s)
2022-05-18T03:39:52.8175136Z   test_mp_embedding_params_and_reduce_diff (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18644
2022-05-18T03:39:52.8200381Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18645
2022-05-18T03:39:53.6606197Z dist init r=1, world=2
2022-05-18T03:39:53.6606523Z dist init r=0, world=2
2022-05-18T03:39:53.6813892Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:39:53.6814305Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:39:53.6814929Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:39:53.6815467Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:39:53.6919335Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:39:53.6919845Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:39:53.9225030Z skip: Need at least 2 CUDA devices (1.109s)
2022-05-18T03:39:53.9266149Z   test_mp_embedding_reduce (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18672
2022-05-18T03:39:53.9291332Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18673
2022-05-18T03:39:54.7683594Z dist init r=1, world=2
2022-05-18T03:39:54.7692920Z dist init r=0, world=2
2022-05-18T03:39:54.7902297Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:39:54.7903213Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:39:54.7903934Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:39:54.7904474Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:39:54.8007795Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:39:54.8008395Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:39:55.0316091Z skip: Need at least 2 CUDA devices (1.109s)
2022-05-18T03:39:55.0357697Z   test_mixed_precision_e2e_full_shard (__main__.TestFSDPMixedPrecisionUnsharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18700
2022-05-18T03:39:55.8716914Z dist init r=0, world=1
2022-05-18T03:39:55.8724703Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:39:55.8725395Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-05-18T03:39:55.8729129Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:39:56.0379055Z skip: Need at least 1 CUDA device (1.006s)
2022-05-18T03:39:56.0420890Z   test_mixed_precision_no_reshard_after_forward (__main__.TestFSDPMixedPrecisionUnsharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18714
2022-05-18T03:39:56.8741403Z dist init r=0, world=1
2022-05-18T03:39:56.8748922Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:39:56.8749582Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-05-18T03:39:56.8752793Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:39:57.0442567Z skip: Need at least 1 CUDA device (1.006s)
2022-05-18T03:39:57.0442889Z 
2022-05-18T03:39:57.0443420Z ----------------------------------------------------------------------
2022-05-18T03:39:57.0443825Z Ran 74 tests in 81.191s
2022-05-18T03:39:57.0443945Z 
2022-05-18T03:39:57.0444019Z OK (skipped=74)
2022-05-18T03:39:57.0444115Z 
2022-05-18T03:39:57.0444203Z Generating XML reports...
2022-05-18T03:39:57.0544102Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_mixed_precision/TEST-TestFSDPMixedPrecisionSharded-20220518033835.xml
2022-05-18T03:39:57.0547426Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_mixed_precision/TEST-TestFSDPMixedPrecisionUnsharded-20220518033835.xml
2022-05-18T03:39:57.2409003Z Running distributed/fsdp/test_fsdp_multiple_forward ... [2022-05-18 03:39:57.240475]
2022-05-18T03:39:57.2409587Z Executing ['/opt/conda/bin/python', 'distributed/fsdp/test_fsdp_multiple_forward.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-05-18 03:39:57.240582]
2022-05-18T03:39:57.8142511Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_multiple_forward
2022-05-18T03:39:57.8154003Z 
2022-05-18T03:39:57.8154431Z Running tests...
2022-05-18T03:39:57.8154828Z ----------------------------------------------------------------------
2022-05-18T03:39:58.0941329Z   test_multi_forward (__main__.TestMultiForward) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18739
2022-05-18T03:39:58.0962486Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18740
2022-05-18T03:39:58.0984870Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18741
2022-05-18T03:39:58.1007690Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 18742
2022-05-18T03:39:58.7322687Z dist init r=1, world=4
2022-05-18T03:39:58.7401912Z dist init r=2, world=4
2022-05-18T03:39:58.7472541Z dist init r=0, world=4
2022-05-18T03:39:58.7599320Z dist init r=3, world=4
2022-05-18T03:39:58.7883763Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:39:58.7985182Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:39:58.7985862Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:39:58.7986536Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:39:58.7987458Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:39:58.7988203Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:39:58.7988777Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:39:58.7989297Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:39:58.7992509Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:39:58.7993282Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:39:58.7993822Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:39:58.7994516Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:39:59.0038283Z skip: Need at least 2 CUDA devices (1.188s)
2022-05-18T03:39:59.0038597Z 
2022-05-18T03:39:59.0039107Z ----------------------------------------------------------------------
2022-05-18T03:39:59.0039365Z Ran 1 test in 1.188s
2022-05-18T03:39:59.0039481Z 
2022-05-18T03:39:59.0039577Z OK (skipped=1)
2022-05-18T03:39:59.0039686Z 
2022-05-18T03:39:59.0039771Z Generating XML reports...
2022-05-18T03:39:59.0074730Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_multiple_forward/TEST-TestMultiForward-20220518033957.xml
2022-05-18T03:39:59.1964812Z Running distributed/fsdp/test_fsdp_multiple_wrapping ... [2022-05-18 03:39:59.196111]
2022-05-18T03:39:59.1965431Z Executing ['/opt/conda/bin/python', 'distributed/fsdp/test_fsdp_multiple_wrapping.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-05-18 03:39:59.196192]
2022-05-18T03:39:59.7640109Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_multiple_wrapping
2022-05-18T03:39:59.7650381Z 
2022-05-18T03:39:59.7650475Z Running tests...
2022-05-18T03:39:59.7651281Z ----------------------------------------------------------------------
2022-05-18T03:39:59.7666267Z   test_multiple_wrapping (__main__.TestMultipleWrapping)
2022-05-18T03:40:00.0467615Z This test simulates wrapping the module after training to run inference. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18806
2022-05-18T03:40:00.0489451Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18807
2022-05-18T03:40:00.0511385Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18808
2022-05-18T03:40:00.0534564Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 18809
2022-05-18T03:40:00.7088931Z dist init r=3, world=4
2022-05-18T03:40:00.7196048Z dist init r=0, world=4
2022-05-18T03:40:00.7350833Z dist init r=1, world=4
2022-05-18T03:40:00.7391544Z dist init r=2, world=4
2022-05-18T03:40:00.7598412Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:40:00.7698556Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:40:00.7801560Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:40:00.7802652Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:40:00.7803340Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:40:00.7803879Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:40:00.7804400Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:40:00.7804920Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:40:00.7907453Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:40:00.7908026Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:40:00.7908564Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:40:00.7909121Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:40:00.9565385Z skip: Need at least 2 CUDA devices (1.191s)
2022-05-18T03:40:00.9565663Z 
2022-05-18T03:40:00.9566155Z ----------------------------------------------------------------------
2022-05-18T03:40:00.9566585Z Ran 1 test in 1.191s
2022-05-18T03:40:00.9566751Z 
2022-05-18T03:40:00.9566826Z OK (skipped=1)
2022-05-18T03:40:00.9566935Z 
2022-05-18T03:40:00.9567021Z Generating XML reports...
2022-05-18T03:40:00.9602410Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_multiple_wrapping/TEST-TestMultipleWrapping-20220518033959.xml
2022-05-18T03:40:01.1551025Z Running distributed/fsdp/test_fsdp_optim_state ... [2022-05-18 03:40:01.154722]
2022-05-18T03:40:01.1551618Z Executing ['/opt/conda/bin/python', 'distributed/fsdp/test_fsdp_optim_state.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-05-18 03:40:01.154805]
2022-05-18T03:40:01.7380917Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_optim_state
2022-05-18T03:40:01.7394550Z 
2022-05-18T03:40:01.7394656Z Running tests...
2022-05-18T03:40:01.7395128Z ----------------------------------------------------------------------
2022-05-18T03:40:01.7408003Z   test_full_optim_state_dict_nested_use_multiple_param_groups_False_rank0_only_False (__main__.TestFSDPOptimState)
2022-05-18T03:40:02.0196936Z Tests :meth:`full_optim_state_dict` by comparing the returned dict for ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18873
2022-05-18T03:40:02.0219015Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18874
2022-05-18T03:40:02.0241497Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18875
2022-05-18T03:40:02.0265804Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 18876
2022-05-18T03:40:02.6268239Z dist init r=0, world=4
2022-05-18T03:40:02.6527049Z dist init r=3, world=4
2022-05-18T03:40:02.6640265Z dist init r=2, world=4
2022-05-18T03:40:02.6799511Z dist init r=1, world=4
2022-05-18T03:40:02.7110505Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:40:02.7212071Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:40:02.7212802Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:40:02.7213312Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:40:02.7214054Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:40:02.7214584Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:40:02.7216490Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:40:02.7217095Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:40:02.7319784Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:40:02.7320366Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:40:02.7320919Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:40:02.7321456Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:40:02.9294881Z skip: Need at least 2 CUDA devices (1.190s)
2022-05-18T03:40:02.9308169Z   test_full_optim_state_dict_nested_use_multiple_param_groups_False_rank0_only_True (__main__.TestFSDPOptimState)
2022-05-18T03:40:02.9345152Z Tests :meth:`full_optim_state_dict` by comparing the returned dict for ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18929
2022-05-18T03:40:02.9371136Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18930
2022-05-18T03:40:02.9394526Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18931
2022-05-18T03:40:02.9418594Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 18932
2022-05-18T03:40:03.5220936Z dist init r=1, world=4
2022-05-18T03:40:03.5385262Z dist init r=0, world=4
2022-05-18T03:40:03.5751427Z dist init r=2, world=4
2022-05-18T03:40:03.5752082Z dist init r=3, world=4
2022-05-18T03:40:03.5861259Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:40:03.5962838Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:40:03.6064280Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:40:03.6064791Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:40:03.6065680Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:40:03.6066304Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:40:03.6066901Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:40:03.6067414Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:40:03.6172046Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:40:03.6172697Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:40:03.6173260Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:40:03.6173675Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:40:03.8444395Z skip: Need at least 2 CUDA devices (0.915s)
2022-05-18T03:40:03.8456961Z   test_full_optim_state_dict_nested_use_multiple_param_groups_True_rank0_only_False (__main__.TestFSDPOptimState)
2022-05-18T03:40:03.8495145Z Tests :meth:`full_optim_state_dict` by comparing the returned dict for ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18985
2022-05-18T03:40:03.8520380Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18986
2022-05-18T03:40:03.8543938Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18987
2022-05-18T03:40:03.8567927Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 18988
2022-05-18T03:40:04.4629118Z dist init r=1, world=4
2022-05-18T03:40:04.4717453Z dist init r=0, world=4
2022-05-18T03:40:04.4906258Z dist init r=3, world=4
2022-05-18T03:40:04.5002891Z dist init r=2, world=4
2022-05-18T03:40:04.5214652Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:40:04.5240819Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:40:04.5342528Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:40:04.5343327Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:40:04.5344310Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:40:04.5344841Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:40:04.5345369Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:40:04.5417996Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:40:04.5451433Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:40:04.5452213Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:40:04.5452570Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:40:04.5452907Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:40:04.7594920Z skip: Need at least 2 CUDA devices (0.915s)
2022-05-18T03:40:04.7607833Z   test_full_optim_state_dict_nested_use_multiple_param_groups_True_rank0_only_True (__main__.TestFSDPOptimState)
2022-05-18T03:40:04.7644109Z Tests :meth:`full_optim_state_dict` by comparing the returned dict for ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19041
2022-05-18T03:40:04.7669749Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19042
2022-05-18T03:40:04.7692924Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19043
2022-05-18T03:40:04.7716679Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 19044
2022-05-18T03:40:05.3525923Z dist init r=0, world=4
2022-05-18T03:40:05.3581500Z dist init r=3, world=4
2022-05-18T03:40:05.3814540Z dist init r=2, world=4
2022-05-18T03:40:05.4205682Z dist init r=1, world=4
2022-05-18T03:40:05.4516962Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:40:05.4617985Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:40:05.4618652Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:40:05.4619297Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:40:05.4619909Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:40:05.4620439Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:40:05.4620955Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:40:05.4621472Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:40:05.4725799Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:40:05.4726503Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:40:05.4727382Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:40:05.4728114Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:40:05.6743055Z skip: Need at least 2 CUDA devices (0.915s)
2022-05-18T03:40:05.6754471Z   test_rekey_optim_state_dict_to_ids_use_multiple_param_groups_False (__main__.TestFSDPOptimState)
2022-05-18T03:40:05.6790417Z Tests :meth:`rekey_optim_state_dict` with the new keys being ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19097
2022-05-18T03:40:05.6817431Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19098
2022-05-18T03:40:05.6841648Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19099
2022-05-18T03:40:05.6871040Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 19100
2022-05-18T03:40:06.3049497Z dist init r=3, world=4
2022-05-18T03:40:06.3095616Z dist init r=1, world=4
2022-05-18T03:40:06.3149135Z dist init r=2, world=4
2022-05-18T03:40:06.3373930Z dist init r=0, world=4
2022-05-18T03:40:06.3708557Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:40:06.3809324Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:40:06.3809958Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:40:06.3810443Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:40:06.3811492Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:40:06.3812202Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:40:06.3812734Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:40:06.3813250Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:40:06.3917442Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:40:06.3918143Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:40:06.3918683Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:40:06.3919173Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:40:06.5899666Z skip: Need at least 2 CUDA devices (0.916s)
2022-05-18T03:40:06.5910824Z   test_rekey_optim_state_dict_to_ids_use_multiple_param_groups_True (__main__.TestFSDPOptimState)
2022-05-18T03:40:06.5946293Z Tests :meth:`rekey_optim_state_dict` with the new keys being ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19153
2022-05-18T03:40:06.5971974Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19154
2022-05-18T03:40:06.5994533Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19155
2022-05-18T03:40:06.6018145Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 19156
2022-05-18T03:40:07.2165246Z dist init r=3, world=4
2022-05-18T03:40:07.2212998Z dist init r=0, world=4
2022-05-18T03:40:07.2250289Z dist init r=2, world=4
2022-05-18T03:40:07.2268341Z dist init r=1, world=4
2022-05-18T03:40:07.2474085Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:40:07.2574044Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:40:07.2662321Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:40:07.2663268Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:40:07.2663996Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:40:07.2664532Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:40:07.2676498Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:40:07.2677030Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:40:07.2783712Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:40:07.2784231Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:40:07.2784761Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:40:07.2785295Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:40:07.5044476Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:40:07.5056077Z   test_rekey_optim_state_dict_to_names_use_multiple_param_groups_False (__main__.TestFSDPOptimState)
2022-05-18T03:40:07.5092618Z Tests :meth:`rekey_optim_state_dict` with the new keys being ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19209
2022-05-18T03:40:07.5117954Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19210
2022-05-18T03:40:07.5140615Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19211
2022-05-18T03:40:07.5164338Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 19212
2022-05-18T03:40:08.0984034Z dist init r=0, world=4
2022-05-18T03:40:08.1116906Z dist init r=2, world=4
2022-05-18T03:40:08.1475127Z dist init r=3, world=4
2022-05-18T03:40:08.1596049Z dist init r=1, world=4
2022-05-18T03:40:08.1784371Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:40:08.1884303Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:40:08.1907403Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:40:08.1908108Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:40:08.1908943Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:40:08.1909454Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:40:08.1986937Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:40:08.1987520Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:40:08.2093001Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:40:08.2093413Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:40:08.2093773Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:40:08.2094284Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:40:08.4191712Z skip: Need at least 2 CUDA devices (0.915s)
2022-05-18T03:40:08.4198920Z   test_scatter_full_optim_state_dict_nested_use_multiple_param_groups_False_wrap_alt_False_halve_world_size_False (__main__.TestFSDPOptimState)
2022-05-18T03:40:08.4234788Z Tests :meth:`scatter_full_optim_state_dict` for a non-FSDP-root ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19265
2022-05-18T03:40:08.4260251Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19266
2022-05-18T03:40:08.4284018Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19267
2022-05-18T03:40:08.4307676Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 19268
2022-05-18T03:40:09.0149147Z dist init r=0, world=4
2022-05-18T03:40:09.0235473Z dist init r=2, world=4
2022-05-18T03:40:09.0677138Z dist init r=3, world=4
2022-05-18T03:40:09.0755829Z dist init r=1, world=4
2022-05-18T03:40:09.1066826Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:40:09.1067412Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:40:09.1168209Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:40:09.1168898Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:40:09.1169875Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:40:09.1170693Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:40:09.1171451Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:40:09.1172164Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:40:09.1176160Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:40:09.1176923Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:40:09.1188987Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:40:09.1190044Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:40:09.3333700Z skip: Need at least 4 CUDA devices (0.914s)
2022-05-18T03:40:09.3341024Z   test_scatter_full_optim_state_dict_nested_use_multiple_param_groups_False_wrap_alt_False_halve_world_size_True (__main__.TestFSDPOptimState)
2022-05-18T03:40:09.3379154Z Tests :meth:`scatter_full_optim_state_dict` for a non-FSDP-root ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19321
2022-05-18T03:40:09.3404226Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19322
2022-05-18T03:40:09.3427056Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19323
2022-05-18T03:40:09.3451205Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 19324
2022-05-18T03:40:09.9548840Z dist init r=2, world=4
2022-05-18T03:40:09.9698981Z dist init r=0, world=4
2022-05-18T03:40:09.9799350Z dist init r=1, world=4
2022-05-18T03:40:09.9808067Z dist init r=3, world=4
2022-05-18T03:40:10.0060578Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:40:10.0161495Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:40:10.0263925Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:40:10.0265022Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:40:10.0265583Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:40:10.0266317Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:40:10.0267165Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:40:10.0268152Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:40:10.0271600Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:40:10.0272272Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:40:10.0272794Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:40:10.0273331Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:40:10.2477761Z skip: Need at least 4 CUDA devices (0.914s)
2022-05-18T03:40:10.2484531Z   test_scatter_full_optim_state_dict_nested_use_multiple_param_groups_False_wrap_alt_True_halve_world_size_False (__main__.TestFSDPOptimState)
2022-05-18T03:40:10.2523108Z Tests :meth:`scatter_full_optim_state_dict` for a non-FSDP-root ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19377
2022-05-18T03:40:10.2548349Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19378
2022-05-18T03:40:10.2571849Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19379
2022-05-18T03:40:10.2596194Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 19380
2022-05-18T03:40:10.8674458Z dist init r=2, world=4
2022-05-18T03:40:10.8769691Z dist init r=1, world=4
2022-05-18T03:40:10.8990903Z dist init r=3, world=4
2022-05-18T03:40:10.9072305Z dist init r=0, world=4
2022-05-18T03:40:10.9482444Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:40:10.9482988Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:40:10.9483493Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:40:10.9483855Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:40:10.9484528Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:40:10.9485106Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:40:10.9485635Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:40:10.9486214Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:40:10.9590543Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:40:10.9591120Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:40:10.9591680Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:40:10.9592047Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:40:11.1622203Z skip: Need at least 4 CUDA devices (0.914s)
2022-05-18T03:40:11.1629437Z   test_scatter_full_optim_state_dict_nested_use_multiple_param_groups_False_wrap_alt_True_halve_world_size_True (__main__.TestFSDPOptimState)
2022-05-18T03:40:11.1664737Z Tests :meth:`scatter_full_optim_state_dict` for a non-FSDP-root ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19433
2022-05-18T03:40:11.1691637Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19434
2022-05-18T03:40:11.1715050Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19435
2022-05-18T03:40:11.1740265Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 19436
2022-05-18T03:40:11.7521853Z dist init r=0, world=4
2022-05-18T03:40:11.7527113Z dist init r=3, world=4
2022-05-18T03:40:11.7694174Z dist init r=1, world=4
2022-05-18T03:40:11.8094015Z dist init r=2, world=4
2022-05-18T03:40:11.8303681Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:40:11.8404341Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:40:11.8506416Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:40:11.8506925Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:40:11.8507776Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:40:11.8508296Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:40:11.8508824Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:40:11.8509351Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:40:11.8613474Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:40:11.8614033Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:40:11.8614925Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:40:11.8616053Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:40:12.0766056Z skip: Need at least 4 CUDA devices (0.914s)
2022-05-18T03:40:12.0773321Z   test_scatter_full_optim_state_dict_nested_use_multiple_param_groups_True_wrap_alt_False_halve_world_size_False (__main__.TestFSDPOptimState)
2022-05-18T03:40:12.0810182Z Tests :meth:`scatter_full_optim_state_dict` for a non-FSDP-root ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19489
2022-05-18T03:40:12.0835796Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19490
2022-05-18T03:40:12.0859028Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19491
2022-05-18T03:40:12.0883296Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 19492
2022-05-18T03:40:12.6876956Z dist init r=3, world=4
2022-05-18T03:40:12.7076164Z dist init r=0, world=4
2022-05-18T03:40:12.7266174Z dist init r=2, world=4
2022-05-18T03:40:12.7392670Z dist init r=1, world=4
2022-05-18T03:40:12.7587002Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:40:12.7778956Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:40:12.7879968Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:40:12.7880904Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:40:12.7881329Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:40:12.7881836Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:40:12.7882352Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:40:12.7891330Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:40:12.7988627Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:40:12.7989324Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:40:12.7989732Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:40:12.7990365Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:40:12.9910403Z skip: Need at least 4 CUDA devices (0.914s)
2022-05-18T03:40:12.9917046Z   test_scatter_full_optim_state_dict_nested_use_multiple_param_groups_True_wrap_alt_False_halve_world_size_True (__main__.TestFSDPOptimState)
2022-05-18T03:40:12.9953765Z Tests :meth:`scatter_full_optim_state_dict` for a non-FSDP-root ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19545
2022-05-18T03:40:12.9979099Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19546
2022-05-18T03:40:13.0002731Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19547
2022-05-18T03:40:13.0026780Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 19548
2022-05-18T03:40:13.6123878Z dist init r=0, world=4
2022-05-18T03:40:13.6225345Z dist init r=3, world=4
2022-05-18T03:40:13.6339938Z dist init r=2, world=4
2022-05-18T03:40:13.6361970Z dist init r=1, world=4
2022-05-18T03:40:13.6649549Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:40:13.6750299Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:40:13.6852493Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:40:13.6853106Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:40:13.6854036Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:40:13.6854766Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:40:13.6855374Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:40:13.6855923Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:40:13.6860678Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:40:13.6861249Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:40:13.6861799Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:40:13.6862300Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:40:13.9053752Z skip: Need at least 4 CUDA devices (0.914s)
2022-05-18T03:40:13.9060252Z   test_scatter_full_optim_state_dict_nested_use_multiple_param_groups_True_wrap_alt_True_halve_world_size_False (__main__.TestFSDPOptimState)
2022-05-18T03:40:13.9097652Z Tests :meth:`scatter_full_optim_state_dict` for a non-FSDP-root ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19601
2022-05-18T03:40:13.9123589Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19602
2022-05-18T03:40:13.9147800Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19603
2022-05-18T03:40:13.9171868Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 19604
2022-05-18T03:40:14.5264944Z dist init r=0, world=4
2022-05-18T03:40:14.5516161Z dist init r=1, world=4
2022-05-18T03:40:14.5552258Z dist init r=2, world=4
2022-05-18T03:40:14.5612708Z dist init r=3, world=4
2022-05-18T03:40:14.5826672Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:40:14.5927658Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:40:14.5928236Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:40:14.5928888Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:40:14.5929779Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:40:14.5930627Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:40:14.5931225Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:40:14.5931747Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:40:14.5939141Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:40:14.5939842Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:40:14.5940352Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:40:14.5943440Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:40:14.8198171Z skip: Need at least 4 CUDA devices (0.914s)
2022-05-18T03:40:14.8204653Z   test_scatter_full_optim_state_dict_nested_use_multiple_param_groups_True_wrap_alt_True_halve_world_size_True (__main__.TestFSDPOptimState)
2022-05-18T03:40:14.8241021Z Tests :meth:`scatter_full_optim_state_dict` for a non-FSDP-root ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19657
2022-05-18T03:40:14.8267585Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19658
2022-05-18T03:40:14.8290520Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19659
2022-05-18T03:40:14.8314659Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 19660
2022-05-18T03:40:15.4064631Z dist init r=2, world=4
2022-05-18T03:40:15.4119857Z dist init r=3, world=4
2022-05-18T03:40:15.4130173Z dist init r=0, world=4
2022-05-18T03:40:15.4635868Z dist init r=1, world=4
2022-05-18T03:40:15.5046208Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:40:15.5046613Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:40:15.5147619Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:40:15.5148329Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:40:15.5149291Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:40:15.5150074Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:40:15.5150635Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:40:15.5151244Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:40:15.5254972Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:40:15.5255618Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:40:15.5256192Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:40:15.5256704Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:40:15.7340960Z skip: Need at least 4 CUDA devices (0.914s)
2022-05-18T03:40:15.7345649Z   test_scatter_full_optim_state_dict_transformer (__main__.TestFSDPOptimState)
2022-05-18T03:40:15.7381484Z Tests :meth:`scatter_full_optim_state_dict` for an FSDP-root ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19713
2022-05-18T03:40:15.7406439Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19714
2022-05-18T03:40:15.7430178Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19715
2022-05-18T03:40:15.7453941Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 19716
2022-05-18T03:40:16.3585226Z dist init r=1, world=4
2022-05-18T03:40:16.3735990Z dist init r=2, world=4
2022-05-18T03:40:16.3877074Z dist init r=3, world=4
2022-05-18T03:40:16.4059776Z dist init r=0, world=4
2022-05-18T03:40:16.4298079Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:40:16.4399224Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:40:16.4500957Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:40:16.4501791Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:40:16.4503064Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:40:16.4503671Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:40:16.4504321Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:40:16.4504950Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:40:16.4509803Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:40:16.4510494Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:40:16.4510998Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:40:16.4511568Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:40:16.6480905Z skip: Need at least 4 CUDA devices (0.914s)
2022-05-18T03:40:16.6487978Z   test_shard_full_optim_state_dict_nested_use_multiple_param_groups_False_wrap_alt_False_halve_world_size_False (__main__.TestFSDPOptimState)
2022-05-18T03:40:16.6524986Z Tests :meth:`shard_full_optim_state_dict` for a non-FSDP-root model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19769
2022-05-18T03:40:16.6551589Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19770
2022-05-18T03:40:16.6574864Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19771
2022-05-18T03:40:16.6598327Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 19772
2022-05-18T03:40:17.2697646Z dist init r=0, world=4
2022-05-18T03:40:17.3040115Z dist init r=2, world=4
2022-05-18T03:40:17.3040442Z dist init r=3, world=4
2022-05-18T03:40:17.3579112Z dist init r=1, world=4
2022-05-18T03:40:17.3914022Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:40:17.3914447Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:40:17.4014999Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:40:17.4016016Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:40:17.4016549Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:40:17.4017181Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:40:17.4017712Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:40:17.4018388Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:40:17.4122382Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:40:17.4123080Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:40:17.4123755Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:40:17.4124488Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:40:17.5624402Z skip: Need at least 4 CUDA devices (0.914s)
2022-05-18T03:40:17.5631008Z   test_shard_full_optim_state_dict_nested_use_multiple_param_groups_False_wrap_alt_False_halve_world_size_True (__main__.TestFSDPOptimState)
2022-05-18T03:40:17.5667048Z Tests :meth:`shard_full_optim_state_dict` for a non-FSDP-root model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19825
2022-05-18T03:40:17.5692611Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19826
2022-05-18T03:40:17.5715381Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19827
2022-05-18T03:40:17.5739501Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 19828
2022-05-18T03:40:18.1708121Z dist init r=2, world=4
2022-05-18T03:40:18.2073377Z dist init r=1, world=4
2022-05-18T03:40:18.2269202Z dist init r=0, world=4
2022-05-18T03:40:18.2357325Z dist init r=3, world=4
2022-05-18T03:40:18.2520194Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:40:18.2620954Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:40:18.2722522Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:40:18.2723407Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:40:18.2724009Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:40:18.2724741Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:40:18.2726961Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:40:18.2727973Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:40:18.2729953Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:40:18.2731076Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:40:18.2731609Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:40:18.2732085Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:40:18.4765795Z skip: Need at least 4 CUDA devices (0.914s)
2022-05-18T03:40:18.4772871Z   test_shard_full_optim_state_dict_nested_use_multiple_param_groups_False_wrap_alt_True_halve_world_size_False (__main__.TestFSDPOptimState)
2022-05-18T03:40:18.4809286Z Tests :meth:`shard_full_optim_state_dict` for a non-FSDP-root model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19881
2022-05-18T03:40:18.4835937Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19882
2022-05-18T03:40:18.4859159Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19883
2022-05-18T03:40:18.4883139Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 19884
2022-05-18T03:40:19.0680229Z dist init r=0, world=4
2022-05-18T03:40:19.0717379Z dist init r=3, world=4
2022-05-18T03:40:19.0874994Z dist init r=1, world=4
2022-05-18T03:40:19.1322627Z dist init r=2, world=4
2022-05-18T03:40:19.1532330Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:40:19.1633852Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:40:19.1634476Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:40:19.1634981Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:40:19.1635786Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:40:19.1636317Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:40:19.1636838Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:40:19.1637362Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:40:19.1739900Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:40:19.1740445Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:40:19.1741008Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:40:19.1741548Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:40:19.3909531Z skip: Need at least 4 CUDA devices (0.914s)
2022-05-18T03:40:19.3916695Z   test_shard_full_optim_state_dict_nested_use_multiple_param_groups_False_wrap_alt_True_halve_world_size_True (__main__.TestFSDPOptimState)
2022-05-18T03:40:19.3952925Z Tests :meth:`shard_full_optim_state_dict` for a non-FSDP-root model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19937
2022-05-18T03:40:19.3979116Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19938
2022-05-18T03:40:19.4003335Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19939
2022-05-18T03:40:19.4027581Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 19940
2022-05-18T03:40:20.0168521Z dist init r=1, world=4
2022-05-18T03:40:20.0273094Z dist init r=0, world=4
2022-05-18T03:40:20.0292328Z dist init r=3, world=4
2022-05-18T03:40:20.0430469Z dist init r=2, world=4
2022-05-18T03:40:20.0601797Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:40:20.0702611Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:40:20.0781857Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:40:20.0782318Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:40:20.0783141Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:40:20.0783718Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:40:20.0804748Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:40:20.0805329Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:40:20.0911664Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:40:20.0912209Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:40:20.0912736Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:40:20.0913455Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:40:20.3053798Z skip: Need at least 4 CUDA devices (0.914s)
2022-05-18T03:40:20.3060813Z   test_shard_full_optim_state_dict_nested_use_multiple_param_groups_True_wrap_alt_False_halve_world_size_False (__main__.TestFSDPOptimState)
2022-05-18T03:40:20.3097042Z Tests :meth:`shard_full_optim_state_dict` for a non-FSDP-root model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19993
2022-05-18T03:40:20.3122131Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19994
2022-05-18T03:40:20.3145389Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19995
2022-05-18T03:40:20.3168914Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 19996
2022-05-18T03:40:20.9397238Z dist init r=2, world=4
2022-05-18T03:40:20.9485196Z dist init r=0, world=4
2022-05-18T03:40:20.9593131Z dist init r=1, world=4
2022-05-18T03:40:20.9709601Z dist init r=3, world=4
2022-05-18T03:40:20.9903601Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:40:21.0005124Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:40:21.0005688Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:40:21.0006300Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:40:21.0006900Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:40:21.0007424Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:40:21.0007945Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:40:21.0009833Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:40:21.0112962Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:40:21.0113669Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:40:21.0114238Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:40:21.0114756Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:40:21.2195341Z skip: Need at least 4 CUDA devices (0.914s)
2022-05-18T03:40:21.2202826Z   test_shard_full_optim_state_dict_nested_use_multiple_param_groups_True_wrap_alt_False_halve_world_size_True (__main__.TestFSDPOptimState)
2022-05-18T03:40:21.2238389Z Tests :meth:`shard_full_optim_state_dict` for a non-FSDP-root model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20049
2022-05-18T03:40:21.2263901Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20050
2022-05-18T03:40:21.2286406Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20051
2022-05-18T03:40:21.2310115Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 20052
2022-05-18T03:40:21.8147056Z dist init r=1, world=4
2022-05-18T03:40:21.8300948Z dist init r=2, world=4
2022-05-18T03:40:21.8714340Z dist init r=3, world=4
2022-05-18T03:40:21.8771992Z dist init r=0, world=4
2022-05-18T03:40:21.9160659Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:40:21.9161333Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:40:21.9262585Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:40:21.9263276Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:40:21.9264285Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:40:21.9264885Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:40:21.9265463Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:40:21.9265989Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:40:21.9369009Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:40:21.9369375Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:40:21.9369864Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:40:21.9370400Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:40:22.1336269Z skip: Need at least 4 CUDA devices (0.914s)
2022-05-18T03:40:22.1343267Z   test_shard_full_optim_state_dict_nested_use_multiple_param_groups_True_wrap_alt_True_halve_world_size_False (__main__.TestFSDPOptimState)
2022-05-18T03:40:22.1379387Z Tests :meth:`shard_full_optim_state_dict` for a non-FSDP-root model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20105
2022-05-18T03:40:22.1403995Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20106
2022-05-18T03:40:22.1427207Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20107
2022-05-18T03:40:22.1450865Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 20108
2022-05-18T03:40:22.7581945Z dist init r=0, world=4
2022-05-18T03:40:22.7717261Z dist init r=1, world=4
2022-05-18T03:40:22.7894125Z dist init r=3, world=4
2022-05-18T03:40:22.7958809Z dist init r=2, world=4
2022-05-18T03:40:22.8228982Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:40:22.8229684Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:40:22.8230288Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:40:22.8231217Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:40:22.8231757Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:40:22.8232242Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:40:22.8232762Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:40:22.8233294Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:40:22.8335985Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:40:22.8336672Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:40:22.8337310Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:40:22.8337831Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:40:23.0477385Z skip: Need at least 4 CUDA devices (0.914s)
2022-05-18T03:40:23.0484731Z   test_shard_full_optim_state_dict_nested_use_multiple_param_groups_True_wrap_alt_True_halve_world_size_True (__main__.TestFSDPOptimState)
2022-05-18T03:40:23.0520125Z Tests :meth:`shard_full_optim_state_dict` for a non-FSDP-root model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20161
2022-05-18T03:40:23.0545634Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20162
2022-05-18T03:40:23.0568805Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20163
2022-05-18T03:40:23.0592580Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 20164
2022-05-18T03:40:23.6275248Z dist init r=3, world=4
2022-05-18T03:40:23.6488946Z dist init r=0, world=4
2022-05-18T03:40:23.6705282Z dist init r=2, world=4
2022-05-18T03:40:23.6888862Z dist init r=1, world=4
2022-05-18T03:40:23.7087604Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:40:23.7188082Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:40:23.7289846Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:40:23.7290442Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:40:23.7291342Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:40:23.7292150Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:40:23.7292942Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:40:23.7293492Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:40:23.7297882Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:40:23.7298467Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:40:23.7299256Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:40:23.7299674Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:40:23.9618662Z skip: Need at least 4 CUDA devices (0.914s)
2022-05-18T03:40:23.9623151Z   test_shard_full_optim_state_dict_transformer (__main__.TestFSDPOptimState)
2022-05-18T03:40:23.9659447Z Tests :meth:`shard_full_optim_state_dict` for an FSDP-root ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20217
2022-05-18T03:40:23.9684911Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20218
2022-05-18T03:40:23.9708022Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20219
2022-05-18T03:40:23.9731344Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 20220
2022-05-18T03:40:24.5816443Z dist init r=2, world=4
2022-05-18T03:40:24.5893544Z dist init r=0, world=4
2022-05-18T03:40:24.6015370Z dist init r=3, world=4
2022-05-18T03:40:24.6095055Z dist init r=1, world=4
2022-05-18T03:40:24.6325731Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:40:24.6326148Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:40:24.6426993Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:40:24.6427697Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:40:24.6428852Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:40:24.6429651Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:40:24.6430285Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:40:24.6431008Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:40:24.6535062Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:40:24.6536203Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:40:24.6536785Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:40:24.6537318Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:40:24.8757710Z skip: Need at least 4 CUDA devices (0.914s)
2022-05-18T03:40:24.8770437Z   test_shard_full_optim_state_dict_unmanaged_params_add_to_fsdp_module_False (__main__.TestFSDPOptimState)
2022-05-18T03:40:24.8806356Z Tests :meth:`shard_full_optim_state_dict` when there are unmanaged ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20273
2022-05-18T03:40:24.8832253Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20274
2022-05-18T03:40:24.8855214Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20275
2022-05-18T03:40:24.8878887Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 20276
2022-05-18T03:40:25.5004705Z dist init r=2, world=4
2022-05-18T03:40:25.5080830Z dist init r=0, world=4
2022-05-18T03:40:25.5220514Z dist init r=1, world=4
2022-05-18T03:40:25.5330230Z dist init r=3, world=4
2022-05-18T03:40:25.5514879Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:40:25.5529956Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:40:25.5631470Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:40:25.5632073Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:40:25.5632842Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:40:25.5633367Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:40:25.5633896Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:40:25.5718051Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:40:25.5739728Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:40:25.5740410Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:40:25.5740809Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:40:25.5741157Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:40:25.7905459Z skip: Need at least 2 CUDA devices (0.915s)
2022-05-18T03:40:25.7918929Z   test_shard_full_optim_state_dict_unmanaged_params_add_to_fsdp_module_True (__main__.TestFSDPOptimState)
2022-05-18T03:40:25.7954836Z Tests :meth:`shard_full_optim_state_dict` when there are unmanaged ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20329
2022-05-18T03:40:25.7980751Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20330
2022-05-18T03:40:25.8004093Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20331
2022-05-18T03:40:25.8028340Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 20332
2022-05-18T03:40:26.4023734Z dist init r=1, world=4
2022-05-18T03:40:26.4483330Z dist init r=2, world=4
2022-05-18T03:40:26.4519305Z dist init r=3, world=4
2022-05-18T03:40:26.4698609Z dist init r=0, world=4
2022-05-18T03:40:26.4931864Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:40:26.5032721Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:40:26.5135451Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:40:26.5136422Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:40:26.5137047Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:40:26.5137862Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:40:26.5138648Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:40:26.5139466Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:40:26.5142746Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:40:26.5143422Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:40:26.5144089Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:40:26.5144741Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:40:26.7054658Z skip: Need at least 2 CUDA devices (0.915s)
2022-05-18T03:40:26.7054941Z 
2022-05-18T03:40:26.7055452Z ----------------------------------------------------------------------
2022-05-18T03:40:26.7055777Z Ran 27 tests in 24.966s
2022-05-18T03:40:26.7055894Z 
2022-05-18T03:40:26.7055969Z OK (skipped=27)
2022-05-18T03:40:26.7056076Z 
2022-05-18T03:40:26.7056150Z Generating XML reports...
2022-05-18T03:40:26.7116724Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_optim_state/TEST-TestFSDPOptimState-20220518034001.xml
2022-05-18T03:40:26.8959901Z Running distributed/fsdp/test_fsdp_overlap ... [2022-05-18 03:40:26.895586]
2022-05-18T03:40:26.8960739Z Executing ['/opt/conda/bin/python', 'distributed/fsdp/test_fsdp_overlap.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-05-18 03:40:26.895666]
2022-05-18T03:40:27.4705392Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_overlap
2022-05-18T03:40:27.4724093Z 
2022-05-18T03:40:27.4724214Z Running tests...
2022-05-18T03:40:27.4724804Z ----------------------------------------------------------------------
2022-05-18T03:40:27.7516395Z   test_forward_overlap (__main__.TestForwardOverlapWorldSizeOne) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20396
2022-05-18T03:40:28.3203341Z dist init r=0, world=1
2022-05-18T03:40:28.3210894Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:40:28.3211958Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-05-18T03:40:28.3215162Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:40:28.4535131Z skip: Need at least 2 CUDA devices (0.981s)
2022-05-18T03:40:28.4543112Z   test_forward_overlap (__main__.TestForwardOverlapWorldSizeTwo) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/71183 for allplatform(s) . If you're seeing this on your local machine and would like to enable this test, please make sure IN_CI is not set and you are not using the flag --import-disabled-tests. (0.001s)
2022-05-18T03:40:28.4543625Z 
2022-05-18T03:40:28.4543837Z ----------------------------------------------------------------------
2022-05-18T03:40:28.4544086Z Ran 2 tests in 0.982s
2022-05-18T03:40:28.4544195Z 
2022-05-18T03:40:28.4544481Z OK (skipped=2)
2022-05-18T03:40:28.4544595Z 
2022-05-18T03:40:28.4544668Z Generating XML reports...
2022-05-18T03:40:28.4579206Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_overlap/TEST-TestForwardOverlapWorldSizeOne-20220518034027.xml
2022-05-18T03:40:28.4581756Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_overlap/TEST-TestForwardOverlapWorldSizeTwo-20220518034027.xml
2022-05-18T03:40:28.6569401Z Running distributed/fsdp/test_fsdp_pure_fp16 ... [2022-05-18 03:40:28.656526]
2022-05-18T03:40:28.6570341Z Executing ['/opt/conda/bin/python', 'distributed/fsdp/test_fsdp_pure_fp16.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-05-18 03:40:28.656607]
2022-05-18T03:40:29.2287861Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_pure_fp16
2022-05-18T03:40:29.2303400Z 
2022-05-18T03:40:29.2303548Z Running tests...
2022-05-18T03:40:29.2304134Z ----------------------------------------------------------------------
2022-05-18T03:40:29.5024810Z   test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) (__main__.TestPureFP16) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/73315 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure IN_CI is not set and you are not using the flag --import-disabled-tests. (0.272s)
2022-05-18T03:40:29.5098488Z   test_pure_fp16_cpu_offload_CPUOffload(offload_params=True) (__main__.TestPureFP16) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20421
2022-05-18T03:40:29.5120486Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20422
2022-05-18T03:40:29.5142541Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20423
2022-05-18T03:40:29.5166241Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 20424
2022-05-18T03:40:30.2076067Z dist init r=3, world=4
2022-05-18T03:40:30.2495776Z dist init r=2, world=4
2022-05-18T03:40:30.2504466Z dist init r=0, world=4
2022-05-18T03:40:30.2678238Z dist init r=1, world=4
2022-05-18T03:40:30.2803527Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:40:30.2888220Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:40:30.2988782Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:40:30.2989464Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:40:30.2990185Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:40:30.2990920Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:40:30.2991454Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:40:30.3006916Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:40:30.3097318Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:40:30.3097982Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:40:30.3098648Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:40:30.3099286Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:40:30.5197811Z skip: Need at least 2 CUDA devices (1.017s)
2022-05-18T03:40:30.5198176Z 
2022-05-18T03:40:30.5198601Z ----------------------------------------------------------------------
2022-05-18T03:40:30.5198852Z Ran 2 tests in 1.289s
2022-05-18T03:40:30.5199232Z 
2022-05-18T03:40:30.5199305Z OK (skipped=2)
2022-05-18T03:40:30.5199400Z 
2022-05-18T03:40:30.5199486Z Generating XML reports...
2022-05-18T03:40:30.5235710Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_pure_fp16/TEST-TestPureFP16-20220518034029.xml
2022-05-18T03:40:30.7171486Z Running distributed/fsdp/test_fsdp_sharded_grad_scaler ... [2022-05-18 03:40:30.716800]
2022-05-18T03:40:30.7172249Z Executing ['/opt/conda/bin/python', 'distributed/fsdp/test_fsdp_sharded_grad_scaler.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-05-18 03:40:30.716883]
2022-05-18T03:40:31.2898569Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_sharded_grad_scaler
2022-05-18T03:40:31.2911431Z 
2022-05-18T03:40:31.2911615Z Running tests...
2022-05-18T03:40:31.2912006Z ----------------------------------------------------------------------
2022-05-18T03:40:31.2921957Z   test_grad_scaling (__main__.TestShardGradScaler) ... skip: no supported device (cuda, xla) found (0.001s)
2022-05-18T03:40:31.2929033Z   test_inf_gradients_skip_optim_step (__main__.TestShardGradScaler) ... skip: no supported device (cuda, xla) found (0.001s)
2022-05-18T03:40:31.2950351Z   test_scaling_unscaling_sparse (__main__.TestShardGradScaler) ... skip: no supported device (cuda, xla) found (0.002s)
2022-05-18T03:40:31.5741366Z   test_scaler_enabled_offload_false_none_mixed_precision (__main__.TestShardedGradScalerParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20488
2022-05-18T03:40:31.5763127Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20489
2022-05-18T03:40:31.5785228Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20490
2022-05-18T03:40:31.5808570Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 20491
2022-05-18T03:40:32.2346560Z dist init r=2, world=4
2022-05-18T03:40:32.2576266Z dist init r=0, world=4
2022-05-18T03:40:32.2813730Z dist init r=1, world=4
2022-05-18T03:40:32.3028171Z dist init r=3, world=4
2022-05-18T03:40:32.3160238Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:40:32.3261288Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:40:32.3363904Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:40:32.3364542Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:40:32.3365373Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:40:32.3366145Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:40:32.3366771Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:40:32.3367304Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:40:32.3370943Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:40:32.3371493Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:40:32.3372010Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:40:32.3375662Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:40:32.5840860Z skip: Need at least 2 CUDA devices (1.289s)
2022-05-18T03:40:32.5883735Z   test_scaler_enabled_offload_false_none_none (__main__.TestShardedGradScalerParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20544
2022-05-18T03:40:32.5910530Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20545
2022-05-18T03:40:32.5933244Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20546
2022-05-18T03:40:32.5958004Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 20547
2022-05-18T03:40:33.1889347Z dist init r=1, world=4
2022-05-18T03:40:33.1984957Z dist init r=0, world=4
2022-05-18T03:40:33.2205833Z dist init r=3, world=4
2022-05-18T03:40:33.2255816Z dist init r=2, world=4
2022-05-18T03:40:33.2414364Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:40:33.2496515Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:40:33.2497399Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:40:33.2497820Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:40:33.2498294Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:40:33.2498807Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:40:33.2499355Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:40:33.2516762Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:40:33.2604800Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:40:33.2605357Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:40:33.2605894Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:40:33.2606453Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:40:33.3982303Z skip: Need at least 2 CUDA devices (0.814s)
2022-05-18T03:40:33.4024658Z   test_scaler_enabled_offload_false_shard_grad_op_mixed_precision (__main__.TestShardedGradScalerParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20600
2022-05-18T03:40:33.4050136Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20601
2022-05-18T03:40:33.4073359Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20602
2022-05-18T03:40:33.4096683Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 20603
2022-05-18T03:40:34.0092905Z dist init r=1, world=4
2022-05-18T03:40:34.0231978Z dist init r=2, world=4
2022-05-18T03:40:34.0437526Z dist init r=0, world=4
2022-05-18T03:40:34.0629666Z dist init r=3, world=4
2022-05-18T03:40:34.0742430Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:40:34.0844371Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:40:34.0844992Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:40:34.0846058Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:40:34.0846888Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:40:34.0847410Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:40:34.0847971Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:40:34.0849069Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:40:34.0854598Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:40:34.0855524Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:40:34.0856718Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:40:34.0857177Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:40:34.3123138Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:40:34.3165938Z   test_scaler_enabled_offload_false_shard_grad_op_none (__main__.TestShardedGradScalerParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20656
2022-05-18T03:40:34.3192542Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20657
2022-05-18T03:40:34.3215488Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20658
2022-05-18T03:40:34.3239384Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 20659
2022-05-18T03:40:34.9178436Z dist init r=0, world=4
2022-05-18T03:40:34.9482697Z dist init r=1, world=4
2022-05-18T03:40:34.9546150Z dist init r=3, world=4
2022-05-18T03:40:34.9676325Z dist init r=2, world=4
2022-05-18T03:40:34.9893274Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:40:34.9994422Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:40:35.0096869Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:40:35.0097443Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:40:35.0098291Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:40:35.0099079Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:40:35.0099867Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:40:35.0100707Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:40:35.0103562Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:40:35.0104104Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:40:35.0104600Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:40:35.0105029Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:40:35.2265673Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:40:35.2308928Z   test_scaler_enabled_offload_true_none_mixed_precision (__main__.TestShardedGradScalerParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20712
2022-05-18T03:40:35.2334206Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20713
2022-05-18T03:40:35.2357568Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20714
2022-05-18T03:40:35.2381491Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 20715
2022-05-18T03:40:35.8141127Z dist init r=0, world=4
2022-05-18T03:40:35.8266318Z dist init r=3, world=4
2022-05-18T03:40:35.8589809Z dist init r=1, world=4
2022-05-18T03:40:35.8710131Z dist init r=2, world=4
2022-05-18T03:40:35.8919755Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:40:35.9000424Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:40:35.9001150Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:40:35.9002168Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:40:35.9002840Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:40:35.9003359Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:40:35.9003886Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:40:35.9021679Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:40:35.9108160Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:40:35.9108551Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:40:35.9108966Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:40:35.9109528Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:40:36.1407978Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:40:36.1451486Z   test_scaler_enabled_offload_true_none_none (__main__.TestShardedGradScalerParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20768
2022-05-18T03:40:36.1477556Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20769
2022-05-18T03:40:36.1500648Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20770
2022-05-18T03:40:36.1524661Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 20771
2022-05-18T03:40:36.7270236Z dist init r=2, world=4
2022-05-18T03:40:36.7514720Z dist init r=1, world=4
2022-05-18T03:40:36.7837816Z dist init r=3, world=4
2022-05-18T03:40:36.7846832Z dist init r=0, world=4
2022-05-18T03:40:36.8045379Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:40:36.8146576Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:40:36.8183152Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:40:36.8183663Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:40:36.8184292Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:40:36.8184825Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:40:36.8249345Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:40:36.8249917Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:40:36.8290470Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:40:36.8291041Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:40:36.8291593Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:40:36.8292047Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:40:37.0551024Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:40:37.0596774Z   test_scaler_enabled_offload_true_shard_grad_op_mixed_precision (__main__.TestShardedGradScalerParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20824
2022-05-18T03:40:37.0622381Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20825
2022-05-18T03:40:37.0646077Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20826
2022-05-18T03:40:37.0670070Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 20827
2022-05-18T03:40:37.6671791Z dist init r=1, world=4
2022-05-18T03:40:37.6672401Z dist init r=2, world=4
2022-05-18T03:40:37.6752233Z dist init r=0, world=4
2022-05-18T03:40:37.6960071Z dist init r=3, world=4
2022-05-18T03:40:37.7163141Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:40:37.7265149Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:40:37.7265777Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:40:37.7268083Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:40:37.7268738Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:40:37.7269462Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:40:37.7270266Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:40:37.7271083Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:40:37.7273995Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:40:37.7274664Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:40:37.7275313Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:40:37.7275919Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:40:37.9696759Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:40:37.9740527Z   test_scaler_enabled_offload_true_shard_grad_op_none (__main__.TestShardedGradScalerParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20880
2022-05-18T03:40:37.9765819Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20881
2022-05-18T03:40:37.9788643Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20882
2022-05-18T03:40:37.9812547Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 20883
2022-05-18T03:40:38.6550452Z dist init r=1, world=4
2022-05-18T03:40:38.6550824Z dist init r=3, world=4
2022-05-18T03:40:38.6785797Z dist init r=0, world=4
2022-05-18T03:40:38.7188940Z dist init r=2, world=4
2022-05-18T03:40:38.7362125Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:40:38.7463159Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:40:38.7564493Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:40:38.7565478Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:40:38.7566384Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:40:38.7567036Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:40:38.7567704Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:40:38.7568437Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:40:38.7572490Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:40:38.7573173Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:40:38.7573918Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:40:38.7574931Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:40:38.9841168Z skip: Need at least 2 CUDA devices (1.014s)
2022-05-18T03:40:38.9841440Z 
2022-05-18T03:40:38.9841858Z ----------------------------------------------------------------------
2022-05-18T03:40:38.9842216Z Ran 11 tests in 7.693s
2022-05-18T03:40:38.9842390Z 
2022-05-18T03:40:38.9842495Z OK (skipped=11)
2022-05-18T03:40:38.9842649Z 
2022-05-18T03:40:38.9842802Z Generating XML reports...
2022-05-18T03:40:38.9878001Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_sharded_grad_scaler/TEST-TestShardGradScaler-20220518034031.xml
2022-05-18T03:40:38.9888290Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_sharded_grad_scaler/TEST-TestShardedGradScalerParityWithDDP-20220518034031.xml
2022-05-18T03:40:39.1788909Z Running distributed/fsdp/test_fsdp_state_dict ... [2022-05-18 03:40:39.178515]
2022-05-18T03:40:39.1789559Z Executing ['/opt/conda/bin/python', 'distributed/fsdp/test_fsdp_state_dict.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-05-18 03:40:39.178594]
2022-05-18T03:40:39.7535707Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_state_dict
2022-05-18T03:40:39.7550890Z 
2022-05-18T03:40:39.7551267Z Running tests...
2022-05-18T03:40:39.7551698Z ----------------------------------------------------------------------
2022-05-18T03:40:39.7567336Z   test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_False_state_dict_rank0_and_offload_False (__main__.TestFSDPStateDict)
2022-05-18T03:40:40.0372673Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20947
2022-05-18T03:40:40.0394702Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20948
2022-05-18T03:40:40.6173356Z dist init r=0, world=2
2022-05-18T03:40:40.6175553Z dist init r=1, world=2
2022-05-18T03:40:40.6382188Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:40:40.6382729Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:40:40.6383679Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:40:40.6384202Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:40:40.6387735Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:40:40.6388403Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:40:40.8419437Z skip: Need at least 2 CUDA devices (1.087s)
2022-05-18T03:40:40.8436746Z   test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_False_state_dict_rank0_and_offload_True (__main__.TestFSDPStateDict)
2022-05-18T03:40:40.8474360Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20975
2022-05-18T03:40:40.8499890Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20976
2022-05-18T03:40:41.4293434Z dist init r=1, world=2
2022-05-18T03:40:41.4563504Z dist init r=0, world=2
2022-05-18T03:40:41.4774570Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:40:41.4774957Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:40:41.4775587Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:40:41.4776391Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:40:41.4779335Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:40:41.4779962Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:40:41.6522697Z skip: Need at least 2 CUDA devices (0.810s)
2022-05-18T03:40:41.6538437Z   test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_True_state_dict_rank0_and_offload_False (__main__.TestFSDPStateDict)
2022-05-18T03:40:41.6573782Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21003
2022-05-18T03:40:41.6598872Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21004
2022-05-18T03:40:42.2318722Z dist init r=1, world=2
2022-05-18T03:40:42.2368236Z dist init r=0, world=2
2022-05-18T03:40:42.2526591Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:40:42.2527028Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:40:42.2527649Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:40:42.2528284Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:40:42.2532641Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:40:42.2533218Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:40:42.4620076Z skip: Need at least 2 CUDA devices (0.810s)
2022-05-18T03:40:42.4636236Z   test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_True_state_dict_rank0_and_offload_True (__main__.TestFSDPStateDict)
2022-05-18T03:40:42.4672333Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21031
2022-05-18T03:40:42.4698954Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21032
2022-05-18T03:40:43.0527557Z dist init r=0, world=2
2022-05-18T03:40:43.0540895Z dist init r=1, world=2
2022-05-18T03:40:43.0648632Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:40:43.0649057Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:40:43.0649658Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:40:43.0650204Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:40:43.0755374Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:40:43.0755938Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:40:43.2720675Z skip: Need at least 2 CUDA devices (0.810s)
2022-05-18T03:40:43.2736249Z   test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_False_state_dict_rank0_and_offload_False (__main__.TestFSDPStateDict)
2022-05-18T03:40:43.2773293Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21059
2022-05-18T03:40:43.2799633Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21060
2022-05-18T03:40:43.8535172Z dist init r=0, world=2
2022-05-18T03:40:43.8535577Z dist init r=1, world=2
2022-05-18T03:40:43.8642601Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:40:43.8645414Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:40:43.8646136Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:40:43.8745115Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:40:43.8749822Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:40:43.8750189Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:40:44.0821070Z skip: Need at least 2 CUDA devices (0.810s)
2022-05-18T03:40:44.0838018Z   test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_False_state_dict_rank0_and_offload_True (__main__.TestFSDPStateDict)
2022-05-18T03:40:44.0874875Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21087
2022-05-18T03:40:44.0900429Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21088
2022-05-18T03:40:44.6640787Z dist init r=0, world=2
2022-05-18T03:40:44.7054230Z dist init r=1, world=2
2022-05-18T03:40:44.7251735Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:40:44.7252193Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:40:44.7252835Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:40:44.7253372Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:40:44.7357370Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:40:44.7357765Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:40:44.8921391Z skip: Need at least 2 CUDA devices (0.810s)
2022-05-18T03:40:44.8937736Z   test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_True_state_dict_rank0_and_offload_False (__main__.TestFSDPStateDict)
2022-05-18T03:40:44.8975647Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21115
2022-05-18T03:40:44.9001065Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21116
2022-05-18T03:40:45.4758206Z dist init r=0, world=2
2022-05-18T03:40:45.4787988Z dist init r=1, world=2
2022-05-18T03:40:45.4896386Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:40:45.4897093Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:40:45.4897731Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:40:45.4898247Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:40:45.5002979Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:40:45.5003547Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:40:45.7021424Z skip: Need at least 2 CUDA devices (0.810s)
2022-05-18T03:40:45.7038381Z   test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_True_state_dict_rank0_and_offload_True (__main__.TestFSDPStateDict)
2022-05-18T03:40:45.7073903Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21143
2022-05-18T03:40:45.7098729Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21144
2022-05-18T03:40:46.2959612Z dist init r=0, world=2
2022-05-18T03:40:46.3047867Z dist init r=1, world=2
2022-05-18T03:40:46.3156184Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:40:46.3156596Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:40:46.3157411Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:40:46.3157941Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:40:46.3262098Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:40:46.3262473Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:40:46.5119061Z skip: Need at least 2 CUDA devices (0.810s)
2022-05-18T03:40:46.5135205Z   test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_False_state_dict_rank0_and_offload_False (__main__.TestFSDPStateDict)
2022-05-18T03:40:46.5171826Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21171
2022-05-18T03:40:46.5197727Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21172
2022-05-18T03:40:47.0937425Z dist init r=1, world=2
2022-05-18T03:40:47.0948521Z dist init r=0, world=2
2022-05-18T03:40:47.1157369Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:40:47.1158126Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:40:47.1158788Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:40:47.1159307Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:40:47.1165742Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:40:47.1166317Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:40:47.3218771Z skip: Need at least 2 CUDA devices (0.810s)
2022-05-18T03:40:47.3234971Z   test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_False_state_dict_rank0_and_offload_True (__main__.TestFSDPStateDict)
2022-05-18T03:40:47.3271761Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21199
2022-05-18T03:40:47.3298332Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21200
2022-05-18T03:40:47.9129974Z dist init r=1, world=2
2022-05-18T03:40:47.9130186Z dist init r=0, world=2
2022-05-18T03:40:47.9339051Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:40:47.9339565Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:40:47.9340188Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:40:47.9340715Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:40:47.9444002Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:40:47.9444806Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:40:48.1319137Z skip: Need at least 2 CUDA devices (0.810s)
2022-05-18T03:40:48.1335567Z   test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_True_state_dict_rank0_and_offload_False (__main__.TestFSDPStateDict)
2022-05-18T03:40:48.1371293Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21227
2022-05-18T03:40:48.1397000Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21228
2022-05-18T03:40:48.7122891Z dist init r=1, world=2
2022-05-18T03:40:48.7143803Z dist init r=0, world=2
2022-05-18T03:40:48.7353515Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:40:48.7354108Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:40:48.7354728Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:40:48.7355262Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:40:48.7458941Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:40:48.7459416Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:40:48.9419222Z skip: Need at least 2 CUDA devices (0.810s)
2022-05-18T03:40:48.9435230Z   test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_True_state_dict_rank0_and_offload_True (__main__.TestFSDPStateDict)
2022-05-18T03:40:48.9473006Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21255
2022-05-18T03:40:48.9498122Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21256
2022-05-18T03:40:49.5246352Z dist init r=1, world=2
2022-05-18T03:40:49.5250923Z dist init r=0, world=2
2022-05-18T03:40:49.5454173Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:40:49.5454824Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:40:49.5455756Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:40:49.5456386Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:40:49.5460306Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:40:49.5460651Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:40:49.7521087Z skip: Need at least 2 CUDA devices (0.810s)
2022-05-18T03:40:49.7536925Z   test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_False_state_dict_rank0_and_offload_False (__main__.TestFSDPStateDict)
2022-05-18T03:40:49.7573829Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21283
2022-05-18T03:40:49.7600332Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21284
2022-05-18T03:40:50.3471426Z dist init r=1, world=2
2022-05-18T03:40:50.3522855Z dist init r=0, world=2
2022-05-18T03:40:50.3732093Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:40:50.3732730Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:40:50.3733615Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:40:50.3734787Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:40:50.3737544Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:40:50.3737897Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:40:50.5623217Z skip: Need at least 2 CUDA devices (0.810s)
2022-05-18T03:40:50.5639104Z   test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_False_state_dict_rank0_and_offload_True (__main__.TestFSDPStateDict)
2022-05-18T03:40:50.5675742Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21311
2022-05-18T03:40:50.5701476Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21312
2022-05-18T03:40:51.1470474Z dist init r=1, world=2
2022-05-18T03:40:51.1743968Z dist init r=0, world=2
2022-05-18T03:40:51.1951931Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:40:51.1952672Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:40:51.1953486Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:40:51.1954020Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:40:51.1957671Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:40:51.1958317Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:40:51.3723027Z skip: Need at least 2 CUDA devices (0.810s)
2022-05-18T03:40:51.3738974Z   test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_True_state_dict_rank0_and_offload_False (__main__.TestFSDPStateDict)
2022-05-18T03:40:51.3775606Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21339
2022-05-18T03:40:51.3800890Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21340
2022-05-18T03:40:51.9541647Z dist init r=0, world=2
2022-05-18T03:40:51.9575292Z dist init r=1, world=2
2022-05-18T03:40:51.9750490Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:40:51.9750926Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:40:51.9751547Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:40:51.9752093Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:40:51.9855919Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:40:51.9856459Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:40:52.1822327Z skip: Need at least 2 CUDA devices (0.810s)
2022-05-18T03:40:52.1839039Z   test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_True_state_dict_rank0_and_offload_True (__main__.TestFSDPStateDict)
2022-05-18T03:40:52.1875330Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21367
2022-05-18T03:40:52.1900758Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21368
2022-05-18T03:40:52.7724646Z dist init r=1, world=2
2022-05-18T03:40:52.7974263Z dist init r=0, world=2
2022-05-18T03:40:52.8183530Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:40:52.8184209Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:40:52.8184881Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:40:52.8185406Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:40:52.8189047Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:40:52.8189686Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:40:52.9923177Z skip: Need at least 2 CUDA devices (0.810s)
2022-05-18T03:40:52.9938653Z   test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_False_state_dict_rank0_and_offload_False (__main__.TestFSDPStateDict)
2022-05-18T03:40:52.9975715Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21395
2022-05-18T03:40:53.0001682Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21396
2022-05-18T03:40:53.5753025Z dist init r=0, world=2
2022-05-18T03:40:53.6057445Z dist init r=1, world=2
2022-05-18T03:40:53.6165047Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:40:53.6165668Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:40:53.6166303Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:40:53.6166838Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:40:53.6270672Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:40:53.6271297Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:40:53.8023703Z skip: Need at least 2 CUDA devices (0.810s)
2022-05-18T03:40:53.8040190Z   test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_False_state_dict_rank0_and_offload_True (__main__.TestFSDPStateDict)
2022-05-18T03:40:53.8076484Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21423
2022-05-18T03:40:53.8103052Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21424
2022-05-18T03:40:54.3848605Z dist init r=1, world=2
2022-05-18T03:40:54.3848958Z dist init r=0, world=2
2022-05-18T03:40:54.4160551Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:40:54.4161107Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:40:54.4161833Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:40:54.4162382Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:40:54.4165828Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:40:54.4166224Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:40:54.6124206Z skip: Need at least 2 CUDA devices (0.810s)
2022-05-18T03:40:54.6140759Z   test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_True_state_dict_rank0_and_offload_False (__main__.TestFSDPStateDict)
2022-05-18T03:40:54.6178146Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21451
2022-05-18T03:40:54.6204508Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21452
2022-05-18T03:40:55.1883627Z dist init r=1, world=2
2022-05-18T03:40:55.1966729Z dist init r=0, world=2
2022-05-18T03:40:55.2091922Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:40:55.2092594Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:40:55.2093469Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:40:55.2094294Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:40:55.2097081Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:40:55.2097683Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:40:55.4225587Z skip: Need at least 2 CUDA devices (0.810s)
2022-05-18T03:40:55.4242224Z   test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_True_state_dict_rank0_and_offload_True (__main__.TestFSDPStateDict)
2022-05-18T03:40:55.4278869Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21479
2022-05-18T03:40:55.4305559Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21480
2022-05-18T03:40:56.0029893Z dist init r=0, world=2
2022-05-18T03:40:56.0043450Z dist init r=1, world=2
2022-05-18T03:40:56.0151438Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:40:56.0152117Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:40:56.0152866Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:40:56.0153382Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:40:56.0159814Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:40:56.0160388Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:40:56.2327061Z skip: Need at least 2 CUDA devices (0.810s)
2022-05-18T03:40:56.2343028Z   test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_False_state_dict_rank0_and_offload_False (__main__.TestFSDPStateDict)
2022-05-18T03:40:56.2379786Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21507
2022-05-18T03:40:56.2405981Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21508
2022-05-18T03:40:56.8246024Z dist init r=1, world=2
2022-05-18T03:40:56.8359519Z dist init r=0, world=2
2022-05-18T03:40:56.8555233Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:40:56.8555716Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:40:56.8556336Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:40:56.8556863Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:40:56.8560610Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:40:56.8560954Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:40:57.0427782Z skip: Need at least 2 CUDA devices (0.810s)
2022-05-18T03:40:57.0444446Z   test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_False_state_dict_rank0_and_offload_True (__main__.TestFSDPStateDict)
2022-05-18T03:40:57.0480834Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21535
2022-05-18T03:40:57.0507078Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21536
2022-05-18T03:40:57.6420010Z dist init r=1, world=2
2022-05-18T03:40:57.6499102Z dist init r=0, world=2
2022-05-18T03:40:57.6628643Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:40:57.6629136Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:40:57.6629817Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:40:57.6634007Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:40:57.6634496Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:40:57.6634841Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:40:57.8528776Z skip: Need at least 2 CUDA devices (0.810s)
2022-05-18T03:40:57.8544852Z   test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_True_state_dict_rank0_and_offload_False (__main__.TestFSDPStateDict)
2022-05-18T03:40:57.8581044Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21563
2022-05-18T03:40:57.8607443Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21564
2022-05-18T03:40:58.4390322Z dist init r=1, world=2
2022-05-18T03:40:58.4390562Z dist init r=0, world=2
2022-05-18T03:40:58.4599074Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:40:58.4599518Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:40:58.4600141Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:40:58.4600708Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:40:58.4705143Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:40:58.4705699Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:40:58.6629858Z skip: Need at least 2 CUDA devices (0.810s)
2022-05-18T03:40:58.6645726Z   test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_True_state_dict_rank0_and_offload_True (__main__.TestFSDPStateDict)
2022-05-18T03:40:58.6681523Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21591
2022-05-18T03:40:58.6707131Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21592
2022-05-18T03:40:59.2432843Z dist init r=1, world=2
2022-05-18T03:40:59.2455491Z dist init r=0, world=2
2022-05-18T03:40:59.2640840Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:40:59.2641297Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:40:59.2641933Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:40:59.2642681Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:40:59.2746620Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:40:59.2747134Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:40:59.4727650Z skip: Need at least 2 CUDA devices (0.810s)
2022-05-18T03:40:59.4772137Z   test_fsdp_state_dict_keys_state_dict_type_local_state_dict (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21619
2022-05-18T03:40:59.4797276Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21620
2022-05-18T03:41:00.0659823Z dist init r=0, world=2
2022-05-18T03:41:00.1048642Z dist init r=1, world=2
2022-05-18T03:41:00.1170042Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:41:00.1170777Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:41:00.1171445Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:41:00.1171970Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:41:00.1275605Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:41:00.1276229Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:41:00.2817634Z skip: Need at least 2 CUDA devices (0.809s)
2022-05-18T03:41:00.2860579Z   test_fsdp_state_dict_keys_state_dict_type_sharded_state_dict (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21647
2022-05-18T03:41:00.2886613Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21648
2022-05-18T03:41:00.8546314Z dist init r=0, world=2
2022-05-18T03:41:00.8604171Z dist init r=1, world=2
2022-05-18T03:41:00.8712232Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:41:00.8712810Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:41:00.8713564Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:41:00.8714098Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:41:00.8818771Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:41:00.8819333Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:41:01.0907858Z skip: Need at least 2 CUDA devices (0.809s)
2022-05-18T03:41:01.0952459Z   test_fsdp_state_dict_keys_state_dict_type_state_dict (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21675
2022-05-18T03:41:01.0983901Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21676
2022-05-18T03:41:01.6761497Z dist init r=0, world=2
2022-05-18T03:41:01.7073180Z dist init r=1, world=2
2022-05-18T03:41:01.7180954Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:41:01.7181521Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:41:01.7182237Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:41:01.7182769Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:41:01.7286934Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:41:01.7287637Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:41:01.9005354Z skip: Need at least 2 CUDA devices (0.810s)
2022-05-18T03:41:01.9050643Z   test_fsdp_state_dict_with_activation_checkpoint_checkpoint_wrap_both (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21703
2022-05-18T03:41:01.9076991Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21704
2022-05-18T03:41:02.4869549Z dist init r=0, world=2
2022-05-18T03:41:02.4869780Z dist init r=1, world=2
2022-05-18T03:41:02.5078501Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:41:02.5079558Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:41:02.5080256Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:41:02.5080800Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:41:02.5083955Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:41:02.5084467Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:41:02.7097607Z skip: Need at least 2 CUDA devices (0.809s)
2022-05-18T03:41:02.7142054Z   test_fsdp_state_dict_with_activation_checkpoint_checkpoint_wrap_first (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21731
2022-05-18T03:41:02.7167422Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21732
2022-05-18T03:41:03.3093989Z dist init r=1, world=2
2022-05-18T03:41:03.3429927Z dist init r=0, world=2
2022-05-18T03:41:03.3603856Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:41:03.3604369Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:41:03.3604998Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:41:03.3605771Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:41:03.3609656Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:41:03.3610278Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:41:03.5188200Z skip: Need at least 2 CUDA devices (0.809s)
2022-05-18T03:41:03.5232412Z   test_fsdp_state_dict_with_activation_checkpoint_checkpoint_wrap_second (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21759
2022-05-18T03:41:03.5257100Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21760
2022-05-18T03:41:04.1071735Z dist init r=0, world=2
2022-05-18T03:41:04.1072135Z dist init r=1, world=2
2022-05-18T03:41:04.1281121Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:41:04.1281832Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:41:04.1282924Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:41:04.1283546Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:41:04.1286471Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:41:04.1287020Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:41:04.3279438Z skip: Need at least 2 CUDA devices (0.809s)
2022-05-18T03:41:04.3326012Z   test_load_activation_checkpointed_module (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21787
2022-05-18T03:41:04.3351059Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21788
2022-05-18T03:41:04.9174209Z dist init r=0, world=2
2022-05-18T03:41:04.9464839Z dist init r=1, world=2
2022-05-18T03:41:04.9584324Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:41:04.9585028Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:41:04.9585895Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:41:04.9586426Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:41:04.9690529Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:41:04.9691116Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:41:05.1371536Z skip: Need at least 2 CUDA devices (0.809s)
2022-05-18T03:41:05.1392346Z   test_save_and_load_after_forward_state_dict_state_dict_type_local_state_dict_mixed_precision_False_state_dict_rank0_and_offload_False (__main__.TestFSDPStateDict)
2022-05-18T03:41:05.1427820Z Test that saving after some training results in params being updated as ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21815
2022-05-18T03:41:05.1453775Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21816
2022-05-18T03:41:05.7276604Z dist init r=1, world=2
2022-05-18T03:41:05.7279727Z dist init r=0, world=2
2022-05-18T03:41:05.7484752Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:41:05.7485251Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:41:05.7485916Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:41:05.7486492Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:41:05.7590611Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:41:05.7591183Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:41:05.9475719Z skip: Need at least 2 CUDA devices (0.810s)
2022-05-18T03:41:05.9496329Z   test_save_and_load_after_forward_state_dict_state_dict_type_local_state_dict_mixed_precision_False_state_dict_rank0_and_offload_True (__main__.TestFSDPStateDict)
2022-05-18T03:41:05.9532916Z Test that saving after some training results in params being updated as ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21843
2022-05-18T03:41:05.9558076Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21844
2022-05-18T03:41:06.5290844Z dist init r=1, world=2
2022-05-18T03:41:06.5550945Z dist init r=0, world=2
2022-05-18T03:41:06.5699991Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:41:06.5700523Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:41:06.5701174Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:41:06.5701707Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:41:06.5705904Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:41:06.5706469Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:41:06.7579563Z skip: Need at least 2 CUDA devices (0.810s)
2022-05-18T03:41:06.7600827Z   test_save_and_load_after_forward_state_dict_state_dict_type_local_state_dict_mixed_precision_True_state_dict_rank0_and_offload_False (__main__.TestFSDPStateDict)
2022-05-18T03:41:06.7635976Z Test that saving after some training results in params being updated as ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21871
2022-05-18T03:41:06.7661378Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21872
2022-05-18T03:41:07.3395682Z dist init r=0, world=2
2022-05-18T03:41:07.3397058Z dist init r=1, world=2
2022-05-18T03:41:07.3603871Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:41:07.3604498Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:41:07.3605227Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:41:07.3605767Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:41:07.3609499Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:41:07.5682137Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:41:07.5682681Z skip: Need at least 2 CUDA devices (0.810s)
2022-05-18T03:41:07.5703888Z   test_save_and_load_after_forward_state_dict_state_dict_type_local_state_dict_mixed_precision_True_state_dict_rank0_and_offload_True (__main__.TestFSDPStateDict)
2022-05-18T03:41:07.5741373Z Test that saving after some training results in params being updated as ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21899
2022-05-18T03:41:07.5766964Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21900
2022-05-18T03:41:08.1675168Z dist init r=0, world=2
2022-05-18T03:41:08.1719743Z dist init r=1, world=2
2022-05-18T03:41:08.1882827Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:41:08.1883399Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:41:08.1884071Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:41:08.1884639Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:41:08.1888217Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:41:08.1888590Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:41:08.3787834Z skip: Need at least 2 CUDA devices (0.810s)
2022-05-18T03:41:08.3809221Z   test_save_and_load_after_forward_state_dict_state_dict_type_sharded_state_dict_mixed_precision_False_state_dict_rank0_and_offload_False (__main__.TestFSDPStateDict)
2022-05-18T03:41:08.3844615Z Test that saving after some training results in params being updated as ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21927
2022-05-18T03:41:08.3870095Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21928
2022-05-18T03:41:08.9721560Z dist init r=1, world=2
2022-05-18T03:41:08.9799238Z dist init r=0, world=2
2022-05-18T03:41:09.0008396Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:41:09.0009091Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:41:09.0009860Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:41:09.0010569Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:41:09.0013805Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:41:09.0014448Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:41:09.1891073Z skip: Need at least 2 CUDA devices (0.810s)
2022-05-18T03:41:09.1911565Z   test_save_and_load_after_forward_state_dict_state_dict_type_sharded_state_dict_mixed_precision_False_state_dict_rank0_and_offload_True (__main__.TestFSDPStateDict)
2022-05-18T03:41:09.1947768Z Test that saving after some training results in params being updated as ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21955
2022-05-18T03:41:09.1973760Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21956
2022-05-18T03:41:09.7752025Z dist init r=0, world=2
2022-05-18T03:41:09.7760460Z dist init r=1, world=2
2022-05-18T03:41:09.7960653Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:41:09.7961168Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:41:09.7961783Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:41:09.7962354Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:41:09.8067405Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:41:09.8067985Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:41:09.9996385Z skip: Need at least 2 CUDA devices (0.810s)
2022-05-18T03:41:10.0016176Z   test_save_and_load_after_forward_state_dict_state_dict_type_sharded_state_dict_mixed_precision_True_state_dict_rank0_and_offload_False (__main__.TestFSDPStateDict)
2022-05-18T03:41:10.0053025Z Test that saving after some training results in params being updated as ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21983
2022-05-18T03:41:10.0079587Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21984
2022-05-18T03:41:10.5914792Z dist init r=0, world=2
2022-05-18T03:41:10.6284220Z dist init r=1, world=2
2022-05-18T03:41:10.6425410Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:41:10.6425877Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:41:10.6426508Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:41:10.6427025Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:41:10.6531056Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:41:10.6531446Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:41:10.8100843Z skip: Need at least 2 CUDA devices (0.810s)
2022-05-18T03:41:10.8122053Z   test_save_and_load_after_forward_state_dict_state_dict_type_sharded_state_dict_mixed_precision_True_state_dict_rank0_and_offload_True (__main__.TestFSDPStateDict)
2022-05-18T03:41:10.8158447Z Test that saving after some training results in params being updated as ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22011
2022-05-18T03:41:10.8183061Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22012
2022-05-18T03:41:11.3915629Z dist init r=0, world=2
2022-05-18T03:41:11.3920339Z dist init r=1, world=2
2022-05-18T03:41:11.4123515Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:41:11.4124205Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:41:11.4124904Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:41:11.4125723Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:41:11.4129872Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:41:11.4130421Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:41:11.6205317Z skip: Need at least 2 CUDA devices (0.810s)
2022-05-18T03:41:11.6225613Z   test_save_and_load_after_forward_state_dict_state_dict_type_state_dict_mixed_precision_False_state_dict_rank0_and_offload_False (__main__.TestFSDPStateDict)
2022-05-18T03:41:11.6263234Z Test that saving after some training results in params being updated as ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22039
2022-05-18T03:41:11.6289859Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22040
2022-05-18T03:41:12.2072241Z dist init r=1, world=2
2022-05-18T03:41:12.2373643Z dist init r=0, world=2
2022-05-18T03:41:12.2481410Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:41:12.2483456Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:41:12.2484373Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:41:12.2584117Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:41:12.2589366Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:41:12.2589759Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:41:12.4311904Z skip: Need at least 2 CUDA devices (0.811s)
2022-05-18T03:41:12.4332324Z   test_save_and_load_after_forward_state_dict_state_dict_type_state_dict_mixed_precision_False_state_dict_rank0_and_offload_True (__main__.TestFSDPStateDict)
2022-05-18T03:41:12.4370538Z Test that saving after some training results in params being updated as ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22067
2022-05-18T03:41:12.4396464Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22068
2022-05-18T03:41:13.0279702Z dist init r=1, world=2
2022-05-18T03:41:13.0496340Z dist init r=0, world=2
2022-05-18T03:41:13.0689990Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:41:13.0690534Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:41:13.0691279Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:41:13.0691897Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:41:13.0696234Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:41:13.0696717Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:41:13.2419624Z skip: Need at least 2 CUDA devices (0.811s)
2022-05-18T03:41:13.2441084Z   test_save_and_load_after_forward_state_dict_state_dict_type_state_dict_mixed_precision_True_state_dict_rank0_and_offload_False (__main__.TestFSDPStateDict)
2022-05-18T03:41:13.2478420Z Test that saving after some training results in params being updated as ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22095
2022-05-18T03:41:13.2504545Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22096
2022-05-18T03:41:13.8258322Z dist init r=1, world=2
2022-05-18T03:41:13.8278099Z dist init r=0, world=2
2022-05-18T03:41:13.8467228Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:41:13.8467708Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:41:13.8468383Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:41:13.8468929Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:41:13.8572509Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:41:13.8572888Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:41:14.0527100Z skip: Need at least 2 CUDA devices (0.811s)
2022-05-18T03:41:14.0548162Z   test_save_and_load_after_forward_state_dict_state_dict_type_state_dict_mixed_precision_True_state_dict_rank0_and_offload_True (__main__.TestFSDPStateDict)
2022-05-18T03:41:14.0585780Z Test that saving after some training results in params being updated as ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22123
2022-05-18T03:41:14.0612777Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22124
2022-05-18T03:41:14.6412744Z dist init r=0, world=2
2022-05-18T03:41:14.6752769Z dist init r=1, world=2
2022-05-18T03:41:14.6923327Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:41:14.6923936Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:41:14.6924566Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:41:14.6925141Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:41:14.7028454Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:41:14.7029038Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:41:14.8634796Z skip: Need at least 2 CUDA devices (0.811s)
2022-05-18T03:41:14.8650793Z   test_state_dict_load_into_local_module_state_dict_type_sharded_state_dict_state_dict_rank0_and_offload_False (__main__.TestFSDPStateDict)
2022-05-18T03:41:14.8686872Z Tests that FSDP's state_dict can be loaded into a local model. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22151
2022-05-18T03:41:14.8711902Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22152
2022-05-18T03:41:15.4385264Z dist init r=0, world=2
2022-05-18T03:41:15.4437606Z dist init r=1, world=2
2022-05-18T03:41:15.4593244Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:41:15.4593943Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:41:15.4594558Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:41:15.4595072Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:41:15.4598164Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:41:15.4598797Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:41:15.6732233Z skip: Need at least 2 CUDA devices (0.810s)
2022-05-18T03:41:15.6748788Z   test_state_dict_load_into_local_module_state_dict_type_sharded_state_dict_state_dict_rank0_and_offload_True (__main__.TestFSDPStateDict)
2022-05-18T03:41:15.6784712Z Tests that FSDP's state_dict can be loaded into a local model. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22179
2022-05-18T03:41:15.6809658Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22180
2022-05-18T03:41:16.2625339Z dist init r=0, world=2
2022-05-18T03:41:16.2642642Z dist init r=1, world=2
2022-05-18T03:41:16.2833646Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:41:16.2834370Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:41:16.2835006Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:41:16.2835539Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:41:16.2939454Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:41:16.2939979Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:41:16.4831513Z skip: Need at least 2 CUDA devices (0.810s)
2022-05-18T03:41:16.4847757Z   test_state_dict_load_into_local_module_state_dict_type_state_dict_state_dict_rank0_and_offload_False (__main__.TestFSDPStateDict)
2022-05-18T03:41:16.4884013Z Tests that FSDP's state_dict can be loaded into a local model. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22207
2022-05-18T03:41:16.4909805Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22208
2022-05-18T03:41:17.0595746Z dist init r=0, world=2
2022-05-18T03:41:17.0622026Z dist init r=1, world=2
2022-05-18T03:41:17.0730958Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:41:17.0731680Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:41:17.0732308Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:41:17.0732840Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:41:17.0836318Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:41:17.0836691Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:41:17.2930457Z skip: Need at least 2 CUDA devices (0.810s)
2022-05-18T03:41:17.2946358Z   test_state_dict_load_into_local_module_state_dict_type_state_dict_state_dict_rank0_and_offload_True (__main__.TestFSDPStateDict)
2022-05-18T03:41:17.2984043Z Tests that FSDP's state_dict can be loaded into a local model. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22235
2022-05-18T03:41:17.3010414Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22236
2022-05-18T03:41:17.8919082Z dist init r=0, world=2
2022-05-18T03:41:17.8919438Z dist init r=1, world=2
2022-05-18T03:41:17.9027148Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:41:17.9027997Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:41:17.9028628Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:41:17.9129052Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:41:17.9133653Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:41:17.9135008Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:41:18.1033006Z skip: Need at least 2 CUDA devices (0.810s)
2022-05-18T03:41:18.1082927Z   test_state_dict_rank0_offload_save_load_flow (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22263
2022-05-18T03:41:18.1108362Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22264
2022-05-18T03:41:18.7082243Z dist init r=1, world=2
2022-05-18T03:41:18.7376370Z dist init r=0, world=2
2022-05-18T03:41:18.7585959Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:41:18.7586400Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:41:18.7587164Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:41:18.7587722Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:41:18.7591708Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:41:18.7592078Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:41:18.9129052Z skip: Need at least 2 CUDA devices (0.810s)
2022-05-18T03:41:18.9171657Z   test_state_dict_save_load_flow_state_dict_type_local_state_dict (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22291
2022-05-18T03:41:18.9197839Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22292
2022-05-18T03:41:19.5147493Z dist init r=1, world=2
2022-05-18T03:41:19.5242710Z dist init r=0, world=2
2022-05-18T03:41:19.5451943Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:41:19.5452660Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:41:19.5453510Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:41:19.5454333Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:41:19.5458351Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:41:19.5458916Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:41:19.7219166Z skip: Need at least 2 CUDA devices (0.809s)
2022-05-18T03:41:19.7261839Z   test_state_dict_save_load_flow_state_dict_type_sharded_state_dict (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22319
2022-05-18T03:41:19.7287884Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22320
2022-05-18T03:41:20.3170667Z dist init r=0, world=2
2022-05-18T03:41:20.3636673Z dist init r=1, world=2
2022-05-18T03:41:20.3781743Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:41:20.3782174Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:41:20.3782784Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:41:20.3783492Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:41:20.3887873Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:41:20.3888521Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:41:20.5308311Z skip: Need at least 2 CUDA devices (0.809s)
2022-05-18T03:41:20.5348582Z   test_state_dict_save_load_flow_state_dict_type_state_dict (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22347
2022-05-18T03:41:20.5373714Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22348
2022-05-18T03:41:21.1229189Z dist init r=0, world=2
2022-05-18T03:41:21.1557926Z dist init r=1, world=2
2022-05-18T03:41:21.1740223Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:41:21.1740985Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:41:21.1741609Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:41:21.1742139Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:41:21.1845807Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:41:21.1846384Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:41:21.3394873Z skip: Need at least 2 CUDA devices (0.808s)
2022-05-18T03:41:21.3454313Z   test_state_dict_skip_module_state_dict_type_local_state_dict_double_nest_True (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22375
2022-05-18T03:41:21.3480833Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22376
2022-05-18T03:41:21.9236516Z dist init r=1, world=2
2022-05-18T03:41:21.9272246Z dist init r=0, world=2
2022-05-18T03:41:21.9444848Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:41:21.9445293Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:41:21.9445912Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:41:21.9446456Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:41:21.9550781Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:41:21.9551192Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:41:22.1502863Z skip: Need at least 2 CUDA devices (0.811s)
2022-05-18T03:41:22.1564169Z   test_state_dict_skip_module_state_dict_type_sharded_state_dict_double_nest_True (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22403
2022-05-18T03:41:22.1591057Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22404
2022-05-18T03:41:22.7503184Z dist init r=1, world=2
2022-05-18T03:41:22.7648931Z dist init r=0, world=2
2022-05-18T03:41:22.7811909Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:41:22.7812566Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:41:22.7813197Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:41:22.7813717Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:41:22.7917883Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:41:22.7918468Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:41:22.9612144Z skip: Need at least 2 CUDA devices (0.811s)
2022-05-18T03:41:22.9671084Z   test_state_dict_skip_module_state_dict_type_state_dict_double_nest_True (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22431
2022-05-18T03:41:22.9696729Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22432
2022-05-18T03:41:23.5513634Z dist init r=1, world=2
2022-05-18T03:41:23.5741291Z dist init r=0, world=2
2022-05-18T03:41:23.5924243Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:41:23.5924955Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:41:23.5926072Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:41:23.5926656Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:41:23.5929727Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:41:23.5930105Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:41:23.7718145Z skip: Need at least 2 CUDA devices (0.810s)
2022-05-18T03:41:23.7761910Z   test_state_dict_type (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22459
2022-05-18T03:41:23.7787950Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22460
2022-05-18T03:41:24.3538143Z dist init r=0, world=2
2022-05-18T03:41:24.3551491Z dist init r=1, world=2
2022-05-18T03:41:24.3746120Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:41:24.3746825Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:41:24.3747675Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:41:24.3748210Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:41:24.3751420Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:41:24.3751847Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:41:24.5809037Z skip: Need at least 2 CUDA devices (0.809s)
2022-05-18T03:41:24.5862154Z   test_state_dict_with_ignored_modules (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22487
2022-05-18T03:41:24.5887785Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22488
2022-05-18T03:41:25.1700436Z dist init r=1, world=2
2022-05-18T03:41:25.1753050Z dist init r=0, world=2
2022-05-18T03:41:25.1908886Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:41:25.1909496Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:41:25.1910126Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:41:25.1910680Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:41:25.1916626Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:41:25.1917223Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:41:25.3910403Z skip: Need at least 2 CUDA devices (0.810s)
2022-05-18T03:41:25.3952716Z   test_wrong_state_dict_config (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22515
2022-05-18T03:41:25.3985605Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22516
2022-05-18T03:41:25.9707266Z dist init r=0, world=2
2022-05-18T03:41:25.9716356Z dist init r=1, world=2
2022-05-18T03:41:25.9824441Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:41:25.9825127Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:41:25.9826153Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:41:25.9826692Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:41:25.9930232Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:41:25.9930847Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:41:26.2007924Z skip: Need at least 2 CUDA devices (0.810s)
2022-05-18T03:41:26.2008101Z 
2022-05-18T03:41:26.2008476Z ----------------------------------------------------------------------
2022-05-18T03:41:26.2008746Z Ran 57 tests in 46.446s
2022-05-18T03:41:26.2008850Z 
2022-05-18T03:41:26.2008948Z OK (skipped=57)
2022-05-18T03:41:26.2009094Z 
2022-05-18T03:41:26.2009182Z Generating XML reports...
2022-05-18T03:41:26.2097983Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_state_dict/TEST-TestFSDPStateDict-20220518034039.xml
2022-05-18T03:41:26.4020717Z Running distributed/fsdp/test_fsdp_summon_full_params ... [2022-05-18 03:41:26.401630]
2022-05-18T03:41:26.4021536Z Executing ['/opt/conda/bin/python', 'distributed/fsdp/test_fsdp_summon_full_params.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-05-18 03:41:26.401708]
2022-05-18T03:41:26.9781005Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_summon_full_params
2022-05-18T03:41:26.9798842Z 
2022-05-18T03:41:26.9798960Z Running tests...
2022-05-18T03:41:26.9799398Z ----------------------------------------------------------------------
2022-05-18T03:41:27.2611106Z   test_cannot_summon_full_params_from_backward (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22554
2022-05-18T03:41:27.2633908Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22555
2022-05-18T03:41:27.8401574Z dist init r=0, world=2
2022-05-18T03:41:27.8401909Z dist init r=1, world=2
2022-05-18T03:41:27.8610198Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:41:27.8610787Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:41:27.8611418Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:41:27.8611933Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:41:27.8715644Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:41:27.8716310Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:41:28.0658294Z skip: Need at least 2 CUDA devices (1.086s)
2022-05-18T03:41:28.0703636Z   test_cannot_summon_full_params_from_forward (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22582
2022-05-18T03:41:28.0729218Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22583
2022-05-18T03:41:28.6470109Z dist init r=1, world=2
2022-05-18T03:41:28.6470810Z dist init r=0, world=2
2022-05-18T03:41:28.6681066Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:41:28.6681771Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:41:28.6682393Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:41:28.6682927Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:41:28.6786183Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:41:28.6786948Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:41:28.8750455Z skip: Need at least 2 CUDA devices (0.809s)
2022-05-18T03:41:28.8797584Z   test_named_parameters_buffers_prefix__recurse_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22610
2022-05-18T03:41:28.8823413Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22611
2022-05-18T03:41:29.4548430Z dist init r=1, world=2
2022-05-18T03:41:29.4586576Z dist init r=0, world=2
2022-05-18T03:41:29.4756852Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:41:29.4757254Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:41:29.4757883Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:41:29.4758433Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:41:29.4861687Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:41:29.4862280Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:41:29.6843808Z skip: Need at least 2 CUDA devices (0.809s)
2022-05-18T03:41:29.6894672Z   test_named_parameters_buffers_prefix__recurse_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22638
2022-05-18T03:41:29.6920526Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22639
2022-05-18T03:41:30.2657403Z dist init r=0, world=2
2022-05-18T03:41:30.2664358Z dist init r=1, world=2
2022-05-18T03:41:30.2866254Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:41:30.2866978Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:41:30.2867723Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:41:30.2868552Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:41:30.2871702Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:41:30.2872071Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:41:30.4940756Z skip: Need at least 2 CUDA devices (0.810s)
2022-05-18T03:41:30.4988050Z   test_named_parameters_buffers_prefix_test_prefix_recurse_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22666
2022-05-18T03:41:30.5014304Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22667
2022-05-18T03:41:31.0855874Z dist init r=0, world=2
2022-05-18T03:41:31.0856247Z dist init r=1, world=2
2022-05-18T03:41:31.0963224Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:41:31.0965022Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:41:31.0966112Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:41:31.1064720Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:41:31.1069246Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:41:31.1069618Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:41:31.3035975Z skip: Need at least 2 CUDA devices (0.809s)
2022-05-18T03:41:31.3083366Z   test_named_parameters_buffers_prefix_test_prefix_recurse_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22694
2022-05-18T03:41:31.3110543Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22695
2022-05-18T03:41:31.8958555Z dist init r=1, world=2
2022-05-18T03:41:31.9046356Z dist init r=0, world=2
2022-05-18T03:41:31.9166768Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:41:31.9167510Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:41:31.9168376Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:41:31.9169218Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:41:31.9172072Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:41:31.9172501Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:41:32.1131835Z skip: Need at least 2 CUDA devices (0.810s)
2022-05-18T03:41:32.1183934Z   test_params_are_unflattenned_rank0_only_False_offload_to_cpu_False_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22722
2022-05-18T03:41:32.1210565Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22723
2022-05-18T03:41:32.7048612Z dist init r=0, world=2
2022-05-18T03:41:32.7109039Z dist init r=1, world=2
2022-05-18T03:41:32.7256203Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:41:32.7256690Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:41:32.7257449Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:41:32.7257986Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:41:32.7362777Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:41:32.7363446Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:41:32.9230955Z skip: Need at least 2 CUDA devices (0.810s)
2022-05-18T03:41:32.9282380Z   test_params_are_unflattenned_rank0_only_False_offload_to_cpu_False_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22750
2022-05-18T03:41:32.9308090Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22751
2022-05-18T03:41:33.5032789Z dist init r=0, world=2
2022-05-18T03:41:33.5038758Z dist init r=1, world=2
2022-05-18T03:41:33.5241931Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:41:33.5242420Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:41:33.5243091Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:41:33.5243608Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:41:33.5348059Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:41:33.5348605Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:41:33.7329215Z skip: Need at least 2 CUDA devices (0.810s)
2022-05-18T03:41:33.7380906Z   test_params_are_unflattenned_rank0_only_False_offload_to_cpu_True_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22778
2022-05-18T03:41:33.7406653Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22779
2022-05-18T03:41:34.3225173Z dist init r=0, world=2
2022-05-18T03:41:34.3283083Z dist init r=1, world=2
2022-05-18T03:41:34.3391196Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:41:34.3391904Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:41:34.3392629Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:41:34.3393160Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:41:34.3496865Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:41:34.3497253Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:41:34.5426868Z skip: Need at least 2 CUDA devices (0.810s)
2022-05-18T03:41:34.5480440Z   test_params_are_unflattenned_rank0_only_False_offload_to_cpu_True_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22806
2022-05-18T03:41:34.5506489Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22807
2022-05-18T03:41:35.1180072Z dist init r=1, world=2
2022-05-18T03:41:35.1229771Z dist init r=0, world=2
2022-05-18T03:41:35.1388655Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:41:35.1389396Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:41:35.1390177Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:41:35.1390973Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:41:35.1394816Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:41:35.1395453Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:41:35.2526578Z skip: Need at least 2 CUDA devices (0.710s)
2022-05-18T03:41:35.2577261Z   test_params_are_unflattenned_rank0_only_True_offload_to_cpu_False_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22834
2022-05-18T03:41:35.2602534Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22835
2022-05-18T03:41:35.8399961Z dist init r=0, world=2
2022-05-18T03:41:35.8512813Z dist init r=1, world=2
2022-05-18T03:41:35.8621577Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:41:35.8622060Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:41:35.8622693Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:41:35.8623433Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:41:35.8727509Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:41:35.8728065Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:41:36.0624354Z skip: Need at least 2 CUDA devices (0.810s)
2022-05-18T03:41:36.0675292Z   test_params_are_unflattenned_rank0_only_True_offload_to_cpu_False_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22862
2022-05-18T03:41:36.0700148Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22863
2022-05-18T03:41:36.6532288Z dist init r=0, world=2
2022-05-18T03:41:36.6532915Z dist init r=1, world=2
2022-05-18T03:41:36.6640056Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:41:36.6641167Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:41:36.6642225Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:41:36.6742659Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:41:36.6746713Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:41:36.6747227Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:41:36.8721614Z skip: Need at least 2 CUDA devices (0.810s)
2022-05-18T03:41:36.8774497Z   test_params_are_unflattenned_rank0_only_True_offload_to_cpu_True_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22890
2022-05-18T03:41:36.8800520Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22891
2022-05-18T03:41:37.4560031Z dist init r=0, world=2
2022-05-18T03:41:37.4560224Z dist init r=1, world=2
2022-05-18T03:41:37.4668939Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:41:37.4669931Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:41:37.4670950Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:41:37.4770335Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:41:37.4775425Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:41:37.4776002Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:41:37.6821123Z skip: Need at least 2 CUDA devices (0.810s)
2022-05-18T03:41:37.6873217Z   test_params_are_unflattenned_rank0_only_True_offload_to_cpu_True_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22918
2022-05-18T03:41:37.6899087Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22919
2022-05-18T03:41:38.2581378Z dist init r=0, world=2
2022-05-18T03:41:38.2581589Z dist init r=1, world=2
2022-05-18T03:41:38.2688255Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:41:38.2691238Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:41:38.2691992Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:41:38.2790629Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:41:38.2794700Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:41:38.2795057Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:41:38.3918790Z skip: Need at least 2 CUDA devices (0.710s)
2022-05-18T03:41:38.3967819Z   test_params_count_and_value_rank0_only_False_offload_to_cpu_False_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22946
2022-05-18T03:41:38.3993651Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22947
2022-05-18T03:41:38.9847398Z dist init r=0, world=2
2022-05-18T03:41:38.9972311Z dist init r=1, world=2
2022-05-18T03:41:39.0156388Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:41:39.0156987Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:41:39.0157601Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:41:39.0158129Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:41:39.0261340Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:41:39.0261894Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:41:39.2014511Z skip: Need at least 2 CUDA devices (0.809s)
2022-05-18T03:41:39.2064987Z   test_params_count_and_value_rank0_only_False_offload_to_cpu_False_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22974
2022-05-18T03:41:39.2090895Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22975
2022-05-18T03:41:39.7932433Z dist init r=1, world=2
2022-05-18T03:41:39.8050155Z dist init r=0, world=2
2022-05-18T03:41:39.8241616Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:41:39.8242145Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:41:39.8242800Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:41:39.8243405Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:41:39.8247780Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:41:39.8248241Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:41:40.0112464Z skip: Need at least 2 CUDA devices (0.810s)
2022-05-18T03:41:40.0162758Z   test_params_count_and_value_rank0_only_False_offload_to_cpu_True_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23002
2022-05-18T03:41:40.0188842Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23003
2022-05-18T03:41:40.6019469Z dist init r=0, world=2
2022-05-18T03:41:40.6066895Z dist init r=1, world=2
2022-05-18T03:41:40.6175050Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:41:40.6175462Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:41:40.6176262Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:41:40.6176813Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:41:40.6280727Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:41:40.6281115Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:41:40.8210030Z skip: Need at least 2 CUDA devices (0.810s)
2022-05-18T03:41:40.8261093Z   test_params_count_and_value_rank0_only_False_offload_to_cpu_True_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23030
2022-05-18T03:41:40.8287497Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23031
2022-05-18T03:41:41.4004077Z dist init r=0, world=2
2022-05-18T03:41:41.4013989Z dist init r=1, world=2
2022-05-18T03:41:41.4122391Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:41:41.4123260Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:41:41.4123907Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:41:41.4124428Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:41:41.4228607Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:41:41.4229210Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:41:41.6310091Z skip: Need at least 2 CUDA devices (0.810s)
2022-05-18T03:41:41.6361691Z   test_params_count_and_value_rank0_only_True_offload_to_cpu_False_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23058
2022-05-18T03:41:41.6387723Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23059
2022-05-18T03:41:42.2138641Z dist init r=0, world=2
2022-05-18T03:41:42.2349641Z dist init r=1, world=2
2022-05-18T03:41:42.2548513Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:41:42.2548931Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:41:42.2549544Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:41:42.2550067Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:41:42.2654623Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:41:42.2655171Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:41:42.4408866Z skip: Need at least 2 CUDA devices (0.810s)
2022-05-18T03:41:42.4460041Z   test_params_count_and_value_rank0_only_True_offload_to_cpu_False_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23086
2022-05-18T03:41:42.4485644Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23087
2022-05-18T03:41:43.0275386Z dist init r=0, world=2
2022-05-18T03:41:43.0397918Z dist init r=1, world=2
2022-05-18T03:41:43.0584634Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:41:43.0585090Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:41:43.0585801Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:41:43.0586352Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:41:43.0591173Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:41:43.0591628Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:41:43.2507115Z skip: Need at least 2 CUDA devices (0.810s)
2022-05-18T03:41:43.2557406Z   test_params_count_and_value_rank0_only_True_offload_to_cpu_True_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23114
2022-05-18T03:41:43.2583668Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23115
2022-05-18T03:41:43.8305524Z dist init r=0, world=2
2022-05-18T03:41:43.8321792Z dist init r=1, world=2
2022-05-18T03:41:43.8514816Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:41:43.8515701Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:41:43.8516393Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:41:43.8516931Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:41:43.8620937Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:41:43.8621494Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:41:44.0606348Z skip: Need at least 2 CUDA devices (0.810s)
2022-05-18T03:41:44.0657296Z   test_params_count_and_value_rank0_only_True_offload_to_cpu_True_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23142
2022-05-18T03:41:44.0681762Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23143
2022-05-18T03:41:44.6390842Z dist init r=1, world=2
2022-05-18T03:41:44.6391046Z dist init r=0, world=2
2022-05-18T03:41:44.6497999Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:41:44.6500716Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:41:44.6501509Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:41:44.6600195Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:41:44.6604597Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:41:44.6606244Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:41:44.8701811Z skip: Need at least 2 CUDA devices (0.809s)
2022-05-18T03:41:44.8743378Z   test_raises_rank0_with_writeback (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23170
2022-05-18T03:41:44.8769440Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23171
2022-05-18T03:41:45.4584119Z dist init r=0, world=2
2022-05-18T03:41:45.4692105Z dist init r=1, world=2
2022-05-18T03:41:45.4800691Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:41:45.4801431Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:41:45.4802104Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:41:45.4802639Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:41:45.4905744Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:41:45.4906305Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:41:45.6790191Z skip: Need at least 2 CUDA devices (0.809s)
2022-05-18T03:41:45.6847316Z   test_reshard_outside_forward_backward_iteration_rank0_only_False_offload_to_cpu_False_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23198
2022-05-18T03:41:45.6873686Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23199
2022-05-18T03:41:46.2727230Z dist init r=0, world=2
2022-05-18T03:41:46.2816409Z dist init r=1, world=2
2022-05-18T03:41:46.2924742Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:41:46.2925206Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:41:46.2926012Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:41:46.2926616Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:41:46.3030659Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:41:46.3031033Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:41:46.4894638Z skip: Need at least 2 CUDA devices (0.810s)
2022-05-18T03:41:46.4951135Z   test_reshard_outside_forward_backward_iteration_rank0_only_False_offload_to_cpu_False_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23226
2022-05-18T03:41:46.4976802Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23227
2022-05-18T03:41:47.0853298Z dist init r=0, world=2
2022-05-18T03:41:47.0927602Z dist init r=1, world=2
2022-05-18T03:41:47.1061328Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:41:47.1061859Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:41:47.1062670Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:41:47.1063360Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:41:47.1066789Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:41:47.1067195Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:41:47.2997177Z skip: Need at least 2 CUDA devices (0.810s)
2022-05-18T03:41:47.3054027Z   test_reshard_outside_forward_backward_iteration_rank0_only_False_offload_to_cpu_True_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23254
2022-05-18T03:41:47.3080118Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23255
2022-05-18T03:41:47.8812839Z dist init r=1, world=2
2022-05-18T03:41:47.8835845Z dist init r=0, world=2
2022-05-18T03:41:47.9020956Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:41:47.9021523Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:41:47.9022218Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:41:47.9022734Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:41:47.9026826Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:41:47.9027222Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:41:48.1100405Z skip: Need at least 2 CUDA devices (0.810s)
2022-05-18T03:41:48.1156810Z   test_reshard_outside_forward_backward_iteration_rank0_only_False_offload_to_cpu_True_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23282
2022-05-18T03:41:48.1181987Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23283
2022-05-18T03:41:48.6904870Z dist init r=0, world=2
2022-05-18T03:41:48.6909244Z dist init r=1, world=2
2022-05-18T03:41:48.7017988Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:41:48.7018667Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:41:48.7019401Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:41:48.7020191Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:41:48.7022700Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:41:48.7023399Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:41:48.8201942Z skip: Need at least 2 CUDA devices (0.710s)
2022-05-18T03:41:48.8256370Z   test_reshard_outside_forward_backward_iteration_rank0_only_True_offload_to_cpu_False_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23310
2022-05-18T03:41:48.8281040Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23311
2022-05-18T03:41:49.4127097Z dist init r=0, world=2
2022-05-18T03:41:49.4162841Z dist init r=1, world=2
2022-05-18T03:41:49.4335891Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:41:49.4336626Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:41:49.4337379Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:41:49.4337912Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:41:49.4441603Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:41:49.4442241Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:41:49.6302194Z skip: Need at least 2 CUDA devices (0.810s)
2022-05-18T03:41:49.6358517Z   test_reshard_outside_forward_backward_iteration_rank0_only_True_offload_to_cpu_False_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23338
2022-05-18T03:41:49.6383443Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23339
2022-05-18T03:41:50.2065793Z dist init r=1, world=2
2022-05-18T03:41:50.2121890Z dist init r=0, world=2
2022-05-18T03:41:50.2273428Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:41:50.2273926Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:41:50.2274552Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:41:50.2275200Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:41:50.2278988Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:41:50.2279540Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:41:50.3402634Z skip: Need at least 2 CUDA devices (0.710s)
2022-05-18T03:41:50.3457950Z   test_reshard_outside_forward_backward_iteration_rank0_only_True_offload_to_cpu_True_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23366
2022-05-18T03:41:50.3484364Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23367
2022-05-18T03:41:50.9345744Z dist init r=0, world=2
2022-05-18T03:41:50.9602212Z dist init r=1, world=2
2022-05-18T03:41:50.9755027Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:41:50.9755619Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:41:50.9756328Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:41:50.9757046Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:41:50.9860611Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:41:50.9860983Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:41:51.1505152Z skip: Need at least 2 CUDA devices (0.810s)
2022-05-18T03:41:51.1560936Z   test_reshard_outside_forward_backward_iteration_rank0_only_True_offload_to_cpu_True_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23394
2022-05-18T03:41:51.1585726Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23395
2022-05-18T03:41:51.7620596Z dist init r=0, world=2
2022-05-18T03:41:51.7809288Z dist init r=1, world=2
2022-05-18T03:41:51.7930115Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:41:51.7930809Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:41:51.7931620Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:41:51.7932152Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:41:51.8035313Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:41:51.8035873Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:41:51.9606539Z skip: Need at least 2 CUDA devices (0.810s)
2022-05-18T03:41:51.9651669Z   test_summon_from_non_fsdp (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23422
2022-05-18T03:41:51.9677135Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23423
2022-05-18T03:41:52.5545400Z dist init r=0, world=2
2022-05-18T03:41:52.5552495Z dist init r=1, world=2
2022-05-18T03:41:52.5754692Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:41:52.5755401Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:41:52.5756020Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:41:52.5756554Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:41:52.5759636Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:41:52.5760143Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:41:52.7699614Z skip: Need at least 2 CUDA devices (0.809s)
2022-05-18T03:41:52.7750681Z   test_summon_full_param_recursive_recurse_False_summon_outer_False_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23450
2022-05-18T03:41:52.7777084Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23451
2022-05-18T03:41:53.3600584Z dist init r=0, world=2
2022-05-18T03:41:53.3645280Z dist init r=1, world=2
2022-05-18T03:41:53.3809100Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:41:53.3809773Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:41:53.3810383Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:41:53.3810919Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:41:53.3914356Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:41:53.3914859Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:41:53.5797894Z skip: Need at least 2 CUDA devices (0.810s)
2022-05-18T03:41:53.5849325Z   test_summon_full_param_recursive_recurse_False_summon_outer_False_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23478
2022-05-18T03:41:53.5874826Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23479
2022-05-18T03:41:54.1578289Z dist init r=1, world=2
2022-05-18T03:41:54.1619033Z dist init r=0, world=2
2022-05-18T03:41:54.1828941Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:41:54.1829645Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:41:54.1830450Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:41:54.1830994Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:41:54.1833947Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:41:54.3896187Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:41:54.3896562Z skip: Need at least 2 CUDA devices (0.810s)
2022-05-18T03:41:54.3947415Z   test_summon_full_param_recursive_recurse_False_summon_outer_True_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23506
2022-05-18T03:41:54.3974828Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23507
2022-05-18T03:41:54.9792177Z dist init r=1, world=2
2022-05-18T03:41:54.9890220Z dist init r=0, world=2
2022-05-18T03:41:55.0100154Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:41:55.0100825Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:41:55.0101722Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:41:55.0102469Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:41:55.0105031Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:41:55.0105463Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:41:55.1995639Z skip: Need at least 2 CUDA devices (0.810s)
2022-05-18T03:41:55.2046301Z   test_summon_full_param_recursive_recurse_False_summon_outer_True_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23534
2022-05-18T03:41:55.2072612Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23535
2022-05-18T03:41:55.7837611Z dist init r=1, world=2
2022-05-18T03:41:55.7853827Z dist init r=0, world=2
2022-05-18T03:41:55.8063404Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:41:55.8063943Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:41:55.8064757Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:41:55.8065350Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:41:55.8068886Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:41:55.8069774Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:41:56.0094806Z skip: Need at least 2 CUDA devices (0.810s)
2022-05-18T03:41:56.0146111Z   test_summon_full_param_recursive_recurse_True_summon_outer_False_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23562
2022-05-18T03:41:56.0172095Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23563
2022-05-18T03:41:56.5913704Z dist init r=1, world=2
2022-05-18T03:41:56.5930291Z dist init r=0, world=2
2022-05-18T03:41:56.6139366Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:41:56.6139962Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:41:56.6140740Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:41:56.6141291Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:41:56.6145267Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:41:56.6145854Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:41:56.8193275Z skip: Need at least 2 CUDA devices (0.810s)
2022-05-18T03:41:56.8246165Z   test_summon_full_param_recursive_recurse_True_summon_outer_False_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23590
2022-05-18T03:41:56.8272324Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23591
2022-05-18T03:41:57.4108135Z dist init r=1, world=2
2022-05-18T03:41:57.4178720Z dist init r=0, world=2
2022-05-18T03:41:57.4317504Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:41:57.4318119Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:41:57.4318930Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:41:57.4319479Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:41:57.4324369Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:41:57.4324929Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:41:57.6293868Z skip: Need at least 2 CUDA devices (0.810s)
2022-05-18T03:41:57.6344971Z   test_summon_full_param_recursive_recurse_True_summon_outer_True_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23618
2022-05-18T03:41:57.6370852Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23619
2022-05-18T03:41:58.2086263Z dist init r=1, world=2
2022-05-18T03:41:58.2426994Z dist init r=0, world=2
2022-05-18T03:41:58.2636401Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:41:58.2637112Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:41:58.2638035Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:41:58.2638554Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:41:58.2642226Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:41:58.2642846Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:41:58.4391843Z skip: Need at least 2 CUDA devices (0.810s)
2022-05-18T03:41:58.4443062Z   test_summon_full_param_recursive_recurse_True_summon_outer_True_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23646
2022-05-18T03:41:58.4469356Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23647
2022-05-18T03:41:59.0215734Z dist init r=1, world=2
2022-05-18T03:41:59.0616453Z dist init r=0, world=2
2022-05-18T03:41:59.0825199Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:41:59.0825900Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:41:59.0826808Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:41:59.0827341Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:41:59.0831066Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:41:59.0831659Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:41:59.2489871Z skip: Need at least 2 CUDA devices (0.810s)
2022-05-18T03:41:59.2538657Z   test_summon_full_param_shard_value_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23674
2022-05-18T03:41:59.2564750Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23675
2022-05-18T03:41:59.8295739Z dist init r=0, world=2
2022-05-18T03:41:59.8323758Z dist init r=1, world=2
2022-05-18T03:41:59.8504106Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:41:59.8504594Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:41:59.8505302Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:41:59.8505841Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:41:59.8609849Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:41:59.8610351Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:42:00.0585902Z skip: Need at least 2 CUDA devices (0.809s)
2022-05-18T03:42:00.0634363Z   test_summon_full_param_shard_value_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23702
2022-05-18T03:42:00.0661314Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23703
2022-05-18T03:42:00.6416900Z dist init r=1, world=2
2022-05-18T03:42:00.6417258Z dist init r=0, world=2
2022-05-18T03:42:00.6625720Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:42:00.6626175Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:42:00.6626792Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:42:00.6627324Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:42:00.6730912Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:42:00.6731405Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:42:00.8682624Z skip: Need at least 2 CUDA devices (0.810s)
2022-05-18T03:42:00.8726617Z   test_summon_full_param_writeback_writeback_False_cpu_offload_CPUOffload(offload_params=False)_mixed_precision_False_modify_outer_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23730
2022-05-18T03:42:00.8752316Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23731
2022-05-18T03:42:01.4504855Z dist init r=1, world=2
2022-05-18T03:42:01.4711386Z dist init r=0, world=2
2022-05-18T03:42:01.4914628Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:42:01.4915217Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:42:01.4915935Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:42:01.4916529Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:42:01.4920134Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:42:01.4920518Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:42:01.6772413Z skip: Need at least 2 CUDA devices (0.809s)
2022-05-18T03:42:01.6815722Z   test_summon_full_param_writeback_writeback_False_cpu_offload_CPUOffload(offload_params=False)_mixed_precision_False_modify_outer_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23758
2022-05-18T03:42:01.6841370Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23759
2022-05-18T03:42:02.2804534Z dist init r=0, world=2
2022-05-18T03:42:02.2841125Z dist init r=1, world=2
2022-05-18T03:42:02.2950488Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:42:02.2950894Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:42:02.2951514Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:42:02.2952059Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:42:02.3057138Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:42:02.3057521Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:42:02.4862656Z skip: Need at least 2 CUDA devices (0.809s)
2022-05-18T03:42:02.4915244Z   test_summon_full_param_writeback_writeback_False_cpu_offload_CPUOffload(offload_params=False)_mixed_precision_True_modify_outer_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23786
2022-05-18T03:42:02.4940707Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23787
2022-05-18T03:42:03.0969037Z dist init r=1, world=2
2022-05-18T03:42:03.1122458Z dist init r=0, world=2
2022-05-18T03:42:03.1277379Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:42:03.1278106Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:42:03.1278724Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:42:03.1279255Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:42:03.1383225Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:42:03.1383815Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:42:03.2962815Z skip: Need at least 2 CUDA devices (0.810s)
2022-05-18T03:42:03.3005211Z   test_summon_full_param_writeback_writeback_False_cpu_offload_CPUOffload(offload_params=False)_mixed_precision_True_modify_outer_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23814
2022-05-18T03:42:03.3030876Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23815
2022-05-18T03:42:03.8767462Z dist init r=1, world=2
2022-05-18T03:42:03.8940286Z dist init r=0, world=2
2022-05-18T03:42:03.9149326Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:42:03.9150063Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:42:03.9150743Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:42:03.9151269Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:42:03.9155423Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:42:03.9156039Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:42:04.1052381Z skip: Need at least 2 CUDA devices (0.809s)
2022-05-18T03:42:04.1105601Z   test_summon_full_param_writeback_writeback_False_cpu_offload_CPUOffload(offload_params=True)_mixed_precision_False_modify_outer_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23842
2022-05-18T03:42:04.1130619Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23843
2022-05-18T03:42:04.7066503Z dist init r=1, world=2
2022-05-18T03:42:04.7370281Z dist init r=0, world=2
2022-05-18T03:42:04.7579036Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:42:04.7579695Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:42:04.7580379Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:42:04.7581256Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:42:04.7584704Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:42:04.7585370Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:42:04.9152671Z skip: Need at least 2 CUDA devices (0.810s)
2022-05-18T03:42:04.9196909Z   test_summon_full_param_writeback_writeback_False_cpu_offload_CPUOffload(offload_params=True)_mixed_precision_False_modify_outer_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23870
2022-05-18T03:42:04.9222545Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23871
2022-05-18T03:42:05.4983542Z dist init r=1, world=2
2022-05-18T03:42:05.5011518Z dist init r=0, world=2
2022-05-18T03:42:05.5221210Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:42:05.5221711Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:42:05.5222436Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:42:05.5223115Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:42:05.5226637Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:42:05.5227039Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:42:05.7243518Z skip: Need at least 2 CUDA devices (0.809s)
2022-05-18T03:42:05.7288621Z   test_summon_full_param_writeback_writeback_False_cpu_offload_CPUOffload(offload_params=True)_mixed_precision_True_modify_outer_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23898
2022-05-18T03:42:05.7314668Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23899
2022-05-18T03:42:06.3172744Z dist init r=1, world=2
2022-05-18T03:42:06.3205734Z dist init r=0, world=2
2022-05-18T03:42:06.3380648Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:42:06.3381126Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:42:06.3381838Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:42:06.3382379Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:42:06.3386367Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:42:06.3386767Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:42:06.5335389Z skip: Need at least 2 CUDA devices (0.809s)
2022-05-18T03:42:06.5380351Z   test_summon_full_param_writeback_writeback_False_cpu_offload_CPUOffload(offload_params=True)_mixed_precision_True_modify_outer_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23926
2022-05-18T03:42:06.5405971Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23927
2022-05-18T03:42:07.1154372Z dist init r=1, world=2
2022-05-18T03:42:07.1154729Z dist init r=0, world=2
2022-05-18T03:42:07.1261596Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:42:07.1264341Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:42:07.1265000Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:42:07.1363983Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:42:07.1367888Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:42:07.1368251Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:42:07.3427466Z skip: Need at least 2 CUDA devices (0.809s)
2022-05-18T03:42:07.3471376Z   test_summon_full_param_writeback_writeback_True_cpu_offload_CPUOffload(offload_params=False)_mixed_precision_False_modify_outer_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23954
2022-05-18T03:42:07.3497148Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23955
2022-05-18T03:42:07.9263738Z dist init r=0, world=2
2022-05-18T03:42:07.9272943Z dist init r=1, world=2
2022-05-18T03:42:07.9472242Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:42:07.9472853Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:42:07.9473605Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:42:07.9474206Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:42:07.9478376Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:42:07.9479087Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:42:08.1518032Z skip: Need at least 2 CUDA devices (0.809s)
2022-05-18T03:42:08.1561682Z   test_summon_full_param_writeback_writeback_True_cpu_offload_CPUOffload(offload_params=False)_mixed_precision_False_modify_outer_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23982
2022-05-18T03:42:08.1588193Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23983
2022-05-18T03:42:08.7496285Z dist init r=0, world=2
2022-05-18T03:42:08.7590427Z dist init r=1, world=2
2022-05-18T03:42:08.7699367Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:42:08.7699829Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:42:08.7700439Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:42:08.7701088Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:42:08.7805424Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:42:08.7805971Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:42:08.9608802Z skip: Need at least 2 CUDA devices (0.809s)
2022-05-18T03:42:08.9653334Z   test_summon_full_param_writeback_writeback_True_cpu_offload_CPUOffload(offload_params=False)_mixed_precision_True_modify_outer_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24010
2022-05-18T03:42:08.9680109Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24011
2022-05-18T03:42:09.5553045Z dist init r=1, world=2
2022-05-18T03:42:09.5912221Z dist init r=0, world=2
2022-05-18T03:42:09.6122392Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:42:09.6123097Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:42:09.6123810Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:42:09.6124342Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:42:09.6127326Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:42:09.6127967Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:42:09.7700851Z skip: Need at least 2 CUDA devices (0.809s)
2022-05-18T03:42:09.7745685Z   test_summon_full_param_writeback_writeback_True_cpu_offload_CPUOffload(offload_params=False)_mixed_precision_True_modify_outer_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24038
2022-05-18T03:42:09.7772100Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24039
2022-05-18T03:42:10.3528973Z dist init r=1, world=2
2022-05-18T03:42:10.3569704Z dist init r=0, world=2
2022-05-18T03:42:10.3737119Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:42:10.3737674Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:42:10.3738563Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:42:10.3739382Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:42:10.3742183Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:42:10.3742660Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:42:10.5792758Z skip: Need at least 2 CUDA devices (0.809s)
2022-05-18T03:42:10.5836997Z   test_summon_full_param_writeback_writeback_True_cpu_offload_CPUOffload(offload_params=True)_mixed_precision_False_modify_outer_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24066
2022-05-18T03:42:10.5863149Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24067
2022-05-18T03:42:11.1610874Z dist init r=1, world=2
2022-05-18T03:42:11.1648320Z dist init r=0, world=2
2022-05-18T03:42:11.1820119Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:42:11.1820559Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:42:11.1821230Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:42:11.1821817Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:42:11.1825204Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:42:11.1825621Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:42:11.3882627Z skip: Need at least 2 CUDA devices (0.809s)
2022-05-18T03:42:11.3926776Z   test_summon_full_param_writeback_writeback_True_cpu_offload_CPUOffload(offload_params=True)_mixed_precision_False_modify_outer_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24094
2022-05-18T03:42:11.3953854Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24095
2022-05-18T03:42:11.9912779Z dist init r=1, world=2
2022-05-18T03:42:11.9990864Z dist init r=0, world=2
2022-05-18T03:42:12.0121299Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:42:12.0121945Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:42:12.0122590Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:42:12.0123119Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:42:12.0129131Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:42:12.0129698Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:42:12.1974807Z skip: Need at least 2 CUDA devices (0.809s)
2022-05-18T03:42:12.2019101Z   test_summon_full_param_writeback_writeback_True_cpu_offload_CPUOffload(offload_params=True)_mixed_precision_True_modify_outer_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24122
2022-05-18T03:42:12.2044755Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24123
2022-05-18T03:42:12.8043336Z dist init r=1, world=2
2022-05-18T03:42:12.8138094Z dist init r=0, world=2
2022-05-18T03:42:12.8250806Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:42:12.8251409Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:42:12.8252171Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:42:12.8252765Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:42:12.8256817Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:42:12.8257249Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:42:13.0065778Z skip: Need at least 2 CUDA devices (0.809s)
2022-05-18T03:42:13.0110490Z   test_summon_full_param_writeback_writeback_True_cpu_offload_CPUOffload(offload_params=True)_mixed_precision_True_modify_outer_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24150
2022-05-18T03:42:13.0136919Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24151
2022-05-18T03:42:13.5948029Z dist init r=1, world=2
2022-05-18T03:42:13.6250574Z dist init r=0, world=2
2022-05-18T03:42:13.6459733Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:42:13.6460111Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:42:13.6460944Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:42:13.6461755Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:42:13.6465237Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:42:13.6465684Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:42:13.8158751Z skip: Need at least 2 CUDA devices (0.809s)
2022-05-18T03:42:13.8207149Z   test_summon_full_params_equivalence_rank0_only_False_offload_to_cpu_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24178
2022-05-18T03:42:13.8233680Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24179
2022-05-18T03:42:14.4023716Z dist init r=1, world=2
2022-05-18T03:42:14.4260207Z dist init r=0, world=2
2022-05-18T03:42:14.4469782Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:42:14.4470298Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:42:14.4470922Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:42:14.4471462Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:42:14.4475534Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:42:14.4476087Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:42:14.6255533Z skip: Need at least 2 CUDA devices (0.810s)
2022-05-18T03:42:14.6303449Z   test_summon_full_params_equivalence_rank0_only_False_offload_to_cpu_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24206
2022-05-18T03:42:14.6330241Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24207
2022-05-18T03:42:15.2170221Z dist init r=1, world=2
2022-05-18T03:42:15.2237286Z dist init r=0, world=2
2022-05-18T03:42:15.2378351Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:42:15.2379028Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:42:15.2379632Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:42:15.2380368Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:42:15.2384548Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:42:15.2385218Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:42:15.4352746Z skip: Need at least 2 CUDA devices (0.810s)
2022-05-18T03:42:15.4400538Z   test_summon_full_params_equivalence_rank0_only_True_offload_to_cpu_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24234
2022-05-18T03:42:15.4426788Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24235
2022-05-18T03:42:16.0186204Z dist init r=0, world=2
2022-05-18T03:42:16.0211981Z dist init r=1, world=2
2022-05-18T03:42:16.0320176Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:42:16.0320597Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:42:16.0321208Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:42:16.0321734Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:42:16.0426056Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:42:16.0426459Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:42:16.2448130Z skip: Need at least 2 CUDA devices (0.809s)
2022-05-18T03:42:16.2496040Z   test_summon_full_params_equivalence_rank0_only_True_offload_to_cpu_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24262
2022-05-18T03:42:16.2522340Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24263
2022-05-18T03:42:16.8372111Z dist init r=0, world=2
2022-05-18T03:42:16.8372379Z dist init r=1, world=2
2022-05-18T03:42:16.8479902Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:42:16.8482348Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:42:16.8483159Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:42:16.8582051Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:42:16.8586472Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:42:16.8587083Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:42:17.0542667Z skip: Need at least 2 CUDA devices (0.809s)
2022-05-18T03:42:17.0591112Z   test_summon_full_params_respects_reshard_after_forward_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24290
2022-05-18T03:42:17.0616675Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24291
2022-05-18T03:42:17.6439623Z dist init r=1, world=2
2022-05-18T03:42:17.6443581Z dist init r=0, world=2
2022-05-18T03:42:17.6647442Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:42:17.6648136Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:42:17.6648990Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:42:17.6649518Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:42:17.6653110Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:42:17.6653532Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:42:17.8637090Z skip: Need at least 2 CUDA devices (0.809s)
2022-05-18T03:42:17.8685205Z   test_summon_full_params_respects_reshard_after_forward_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24318
2022-05-18T03:42:17.8711751Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24319
2022-05-18T03:42:18.4477775Z dist init r=1, world=2
2022-05-18T03:42:18.4478023Z dist init r=0, world=2
2022-05-18T03:42:18.4587238Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:42:18.4587748Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:42:18.4588649Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:42:18.4589353Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:42:18.4592182Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:42:18.4592571Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:42:18.5730988Z skip: Need at least 2 CUDA devices (0.709s)
2022-05-18T03:42:18.5775024Z   test_summon_single_param (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24346
2022-05-18T03:42:18.5800643Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24347
2022-05-18T03:42:19.1587773Z dist init r=1, world=2
2022-05-18T03:42:19.1594375Z dist init r=0, world=2
2022-05-18T03:42:19.1796702Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:42:19.1797380Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:42:19.1798006Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:42:19.1798532Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:42:19.1902111Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:42:19.1902714Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:42:19.3821807Z skip: Need at least 2 CUDA devices (0.809s)
2022-05-18T03:42:19.3865890Z   test_summon_full_param_writeback_writeback_False_modify_outer_False_mixed_precision_False (__main__.TestSummonFullParamsNoShard) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24374
2022-05-18T03:42:19.9563608Z dist init r=0, world=1
2022-05-18T03:42:19.9571771Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:42:19.9572741Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-05-18T03:42:19.9576035Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:42:20.0884821Z skip: Need at least 2 CUDA devices (0.706s)
2022-05-18T03:42:20.0927878Z   test_summon_full_param_writeback_writeback_False_modify_outer_False_mixed_precision_True (__main__.TestSummonFullParamsNoShard) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24388
2022-05-18T03:42:20.6630126Z dist init r=0, world=1
2022-05-18T03:42:20.6637942Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:42:20.6638633Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-05-18T03:42:20.6642482Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:42:20.7945960Z skip: Need at least 2 CUDA devices (0.706s)
2022-05-18T03:42:20.7988733Z   test_summon_full_param_writeback_writeback_False_modify_outer_True_mixed_precision_False (__main__.TestSummonFullParamsNoShard) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24402
2022-05-18T03:42:21.3780641Z dist init r=0, world=1
2022-05-18T03:42:21.3789173Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:42:21.3789979Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-05-18T03:42:21.3792998Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:42:21.5007528Z skip: Need at least 2 CUDA devices (0.706s)
2022-05-18T03:42:21.5052354Z   test_summon_full_param_writeback_writeback_False_modify_outer_True_mixed_precision_True (__main__.TestSummonFullParamsNoShard) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24416
2022-05-18T03:42:22.0783639Z dist init r=0, world=1
2022-05-18T03:42:22.0791854Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:42:22.0792829Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-05-18T03:42:22.0796658Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:42:22.2069819Z skip: Need at least 2 CUDA devices (0.706s)
2022-05-18T03:42:22.2112327Z   test_summon_full_param_writeback_writeback_True_modify_outer_False_mixed_precision_False (__main__.TestSummonFullParamsNoShard) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24430
2022-05-18T03:42:22.7838050Z dist init r=0, world=1
2022-05-18T03:42:22.7845498Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:42:22.7846139Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-05-18T03:42:22.7850517Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:42:22.9130452Z skip: Need at least 2 CUDA devices (0.706s)
2022-05-18T03:42:22.9173190Z   test_summon_full_param_writeback_writeback_True_modify_outer_False_mixed_precision_True (__main__.TestSummonFullParamsNoShard) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24444
2022-05-18T03:42:23.4877862Z dist init r=0, world=1
2022-05-18T03:42:23.4886082Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:42:23.4886748Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-05-18T03:42:23.4889954Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:42:23.6193998Z skip: Need at least 2 CUDA devices (0.706s)
2022-05-18T03:42:23.6236226Z   test_summon_full_param_writeback_writeback_True_modify_outer_True_mixed_precision_False (__main__.TestSummonFullParamsNoShard) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24458
2022-05-18T03:42:24.1930714Z dist init r=0, world=1
2022-05-18T03:42:24.1938170Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:42:24.1939293Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-05-18T03:42:24.1943164Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:42:24.3254344Z skip: Need at least 2 CUDA devices (0.706s)
2022-05-18T03:42:24.3296841Z   test_summon_full_param_writeback_writeback_True_modify_outer_True_mixed_precision_True (__main__.TestSummonFullParamsNoShard) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24472
2022-05-18T03:42:24.9003089Z dist init r=0, world=1
2022-05-18T03:42:24.9010227Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:42:24.9010928Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-05-18T03:42:24.9014852Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:42:25.0314926Z skip: Need at least 2 CUDA devices (0.706s)
2022-05-18T03:42:25.0315563Z 
2022-05-18T03:42:25.0316110Z ----------------------------------------------------------------------
2022-05-18T03:42:25.0316422Z Ran 73 tests in 58.052s
2022-05-18T03:42:25.0316541Z 
2022-05-18T03:42:25.0316614Z OK (skipped=73)
2022-05-18T03:42:25.0316723Z 
2022-05-18T03:42:25.0316809Z Generating XML reports...
2022-05-18T03:42:25.0409439Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_summon_full_params/TEST-TestSummonFullParams-20220518034126.xml
2022-05-18T03:42:25.0418441Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_summon_full_params/TEST-TestSummonFullParamsNoShard-20220518034126.xml
2022-05-18T03:42:25.2331298Z Running distributed/fsdp/test_fsdp_traversal ... [2022-05-18 03:42:25.232716]
2022-05-18T03:42:25.2332273Z Executing ['/opt/conda/bin/python', 'distributed/fsdp/test_fsdp_traversal.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-05-18 03:42:25.232800]
2022-05-18T03:42:25.8030992Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_traversal
2022-05-18T03:42:25.8041912Z 
2022-05-18T03:42:25.8042220Z Running tests...
2022-05-18T03:42:25.8042882Z ----------------------------------------------------------------------
2022-05-18T03:42:26.0830087Z   test_fsdp_modules (__main__.TestTraversal) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24497
2022-05-18T03:42:26.0851826Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24498
2022-05-18T03:42:26.6544491Z dist init r=1, world=2
2022-05-18T03:42:26.6544888Z dist init r=0, world=2
2022-05-18T03:42:26.6651663Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:42:26.6653001Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:42:26.6654147Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:42:26.6753301Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:42:26.6757994Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:42:26.6758549Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:42:26.8876509Z skip: Need at least 2 CUDA devices (1.083s)
2022-05-18T03:42:26.8876925Z 
2022-05-18T03:42:26.8877473Z ----------------------------------------------------------------------
2022-05-18T03:42:26.8877780Z Ran 1 test in 1.083s
2022-05-18T03:42:26.8877896Z 
2022-05-18T03:42:26.8877973Z OK (skipped=1)
2022-05-18T03:42:26.8878082Z 
2022-05-18T03:42:26.8878189Z Generating XML reports...
2022-05-18T03:42:26.8911770Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_traversal/TEST-TestTraversal-20220518034225.xml
2022-05-18T03:42:27.0788103Z Running distributed/fsdp/test_fsdp_uneven ... [2022-05-18 03:42:27.078405]
2022-05-18T03:42:27.0788680Z Executing ['/opt/conda/bin/python', 'distributed/fsdp/test_fsdp_uneven.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-05-18 03:42:27.078489]
2022-05-18T03:42:27.6515426Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_uneven
2022-05-18T03:42:27.6525231Z 
2022-05-18T03:42:27.6525379Z Running tests...
2022-05-18T03:42:27.6525855Z ----------------------------------------------------------------------
2022-05-18T03:42:27.6541793Z   test_one_iteration (__main__.TestUnevenParamShard)
2022-05-18T03:42:27.9303707Z Test FSDP with uneven divide of parameter shards. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24536
2022-05-18T03:42:27.9325669Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24537
2022-05-18T03:42:27.9349474Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24538
2022-05-18T03:42:27.9372609Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 24539
2022-05-18T03:42:28.6012100Z dist init r=3, world=4
2022-05-18T03:42:28.6042044Z dist init r=1, world=4
2022-05-18T03:42:28.6156549Z dist init r=0, world=4
2022-05-18T03:42:28.6348666Z dist init r=2, world=4
2022-05-18T03:42:28.6521381Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:42:28.6567400Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:42:28.6568429Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:42:28.6569029Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:42:28.6569734Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:42:28.6570244Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:42:28.6570774Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:42:28.6623457Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:42:28.6675678Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:42:28.6676162Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:42:28.6676665Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:42:28.6677228Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:42:28.8403079Z skip: Need at least 2 CUDA devices (1.187s)
2022-05-18T03:42:28.8403456Z 
2022-05-18T03:42:28.8403972Z ----------------------------------------------------------------------
2022-05-18T03:42:28.8404439Z Ran 1 test in 1.188s
2022-05-18T03:42:28.8404602Z 
2022-05-18T03:42:28.8404676Z OK (skipped=1)
2022-05-18T03:42:28.8404771Z 
2022-05-18T03:42:28.8404858Z Generating XML reports...
2022-05-18T03:42:28.8439139Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_uneven/TEST-TestUnevenParamShard-20220518034227.xml
2022-05-18T03:42:29.0341037Z Running distributed/fsdp/test_shard_utils ... [2022-05-18 03:42:29.033689]
2022-05-18T03:42:29.0341835Z Executing ['/opt/conda/bin/python', 'distributed/fsdp/test_shard_utils.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-05-18 03:42:29.033771]
2022-05-18T03:42:29.6994899Z Running distributed/fsdp/test_utils ... [2022-05-18 03:42:29.699050]
2022-05-18T03:42:29.6995469Z Executing ['/opt/conda/bin/python', 'distributed/fsdp/test_utils.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-05-18 03:42:29.699134]
2022-05-18T03:42:30.2688347Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_utils
2022-05-18T03:42:30.2698002Z 
2022-05-18T03:42:30.2698139Z Running tests...
2022-05-18T03:42:30.2699047Z ----------------------------------------------------------------------
2022-05-18T03:42:30.5460679Z   test_apply_to_tensors_cpu_cuda (__main__.TestUtils) ... skip: Skipped due to lack of GPU (0.276s)
2022-05-18T03:42:30.5488445Z   test_apply_to_tensors_devices_['cpu'] (__main__.TestUtils) ... ok (0.003s)
2022-05-18T03:42:30.5506349Z   test_apply_to_tensors_devices_['cuda'] (__main__.TestUtils) ... skip: Skipped due to lack of GPU (0.002s)
2022-05-18T03:42:30.5512366Z   test_packed_sequence (__main__.TestUtils)
2022-05-18T03:42:30.5530431Z Test to ensure RNN packed sequences are modified correctly. ... ok (0.002s)
2022-05-18T03:42:30.5540362Z   test_replace_by_prefix (__main__.TestUtils) ... ok (0.001s)
2022-05-18T03:42:30.5540638Z 
2022-05-18T03:42:30.5541199Z ----------------------------------------------------------------------
2022-05-18T03:42:30.5541655Z Ran 5 tests in 0.284s
2022-05-18T03:42:30.5541864Z 
2022-05-18T03:42:30.5541969Z OK (skipped=2)
2022-05-18T03:42:30.5542079Z 
2022-05-18T03:42:30.5542167Z Generating XML reports...
2022-05-18T03:42:30.5575057Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_utils/TEST-TestUtils-20220518034230.xml
2022-05-18T03:42:30.7152407Z Running distributed/fsdp/test_wrap ... [2022-05-18 03:42:30.714824]
2022-05-18T03:42:30.7152958Z Executing ['/opt/conda/bin/python', 'distributed/fsdp/test_wrap.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-05-18 03:42:30.714904]
2022-05-18T03:42:31.2924031Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_wrap
2022-05-18T03:42:31.2938266Z 
2022-05-18T03:42:31.2938549Z Running tests...
2022-05-18T03:42:31.2938957Z ----------------------------------------------------------------------
2022-05-18T03:42:31.2944791Z   test_always_wrap (__main__.TestAutoWrap)
2022-05-18T03:42:31.2945635Z Test to ensure that if `always_wrap_policy` is ... skip: Test Requires CUDA (0.001s)
2022-05-18T03:42:31.2957257Z   test_always_wrap_with_ignored_modules_wrap_method_WrapMethod_FSDP_CTOR (__main__.TestAutoWrap) ... skip: Requires at least 2 GPUs (0.001s)
2022-05-18T03:42:31.2968257Z   test_always_wrap_with_ignored_modules_wrap_method_WrapMethod_WRAP_API (__main__.TestAutoWrap) ... skip: Requires at least 2 GPUs (0.001s)
2022-05-18T03:42:31.2972879Z   test_auto_wrap_api (__main__.TestAutoWrap)
2022-05-18T03:42:31.2973377Z Test to ensure with auto wrap, we wrap child modules correctly based on the min_num_params. ... skip: Requires at least 2 GPUs (0.000s)
2022-05-18T03:42:31.2979780Z   test_auto_wrap_preset_exclude_wrap (__main__.TestAutoWrap)
2022-05-18T03:42:31.2980439Z Test to ensure excluded modules are not wrapped, regardless if the total param size is greater than the ... skip: Requires at least 2 GPUs (0.001s)
2022-05-18T03:42:31.2985746Z   test_auto_wrap_preset_exclude_wrap_include_children (__main__.TestAutoWrap)
2022-05-18T03:42:31.2986462Z Test to ensure excluded modules are not wrapped, but children are if param size is greater than ... skip: Requires at least 2 GPUs (0.001s)
2022-05-18T03:42:31.2993089Z   test_auto_wrap_preset_force_leaf (__main__.TestAutoWrap)
2022-05-18T03:42:31.2993625Z Test to ensure force-leaf modules are not wrapped, and children are not wrapped. The ... skip: Requires at least 2 GPUs (0.001s)
2022-05-18T03:42:31.3000642Z   test_auto_wrap_preset_force_leaf_custom (__main__.TestAutoWrap)
2022-05-18T03:42:31.3001127Z Test to ensure force-leaf modules are not wrapped. ... skip: Requires at least 2 GPUs (0.001s)
2022-05-18T03:42:31.3017236Z   test_auto_wrap_smoke_test_fsdp_init_mode_FSDPInitMode_CUDA_AFTER_cpu_offload_CPUOffload(offload_params=False)_use_device_id_False (__main__.TestAutoWrap) ... skip: Test Requires CUDA (0.002s)
2022-05-18T03:42:31.3033262Z   test_auto_wrap_smoke_test_fsdp_init_mode_FSDPInitMode_CUDA_AFTER_cpu_offload_CPUOffload(offload_params=False)_use_device_id_True (__main__.TestAutoWrap) ... skip: Test Requires CUDA (0.002s)
2022-05-18T03:42:31.3049351Z   test_auto_wrap_smoke_test_fsdp_init_mode_FSDPInitMode_CUDA_AFTER_cpu_offload_CPUOffload(offload_params=True)_use_device_id_False (__main__.TestAutoWrap) ... skip: Test Requires CUDA (0.002s)
2022-05-18T03:42:31.3065594Z   test_auto_wrap_smoke_test_fsdp_init_mode_FSDPInitMode_CUDA_AFTER_cpu_offload_CPUOffload(offload_params=True)_use_device_id_True (__main__.TestAutoWrap) ... skip: Test Requires CUDA (0.002s)
2022-05-18T03:42:31.3081821Z   test_auto_wrap_smoke_test_fsdp_init_mode_FSDPInitMode_CUDA_BEFORE_cpu_offload_CPUOffload(offload_params=False)_use_device_id_False (__main__.TestAutoWrap) ... skip: Test Requires CUDA (0.002s)
2022-05-18T03:42:31.3097741Z   test_auto_wrap_smoke_test_fsdp_init_mode_FSDPInitMode_CUDA_BEFORE_cpu_offload_CPUOffload(offload_params=False)_use_device_id_True (__main__.TestAutoWrap) ... skip: Test Requires CUDA (0.002s)
2022-05-18T03:42:31.3114168Z   test_auto_wrap_smoke_test_fsdp_init_mode_FSDPInitMode_CUDA_BEFORE_cpu_offload_CPUOffload(offload_params=True)_use_device_id_False (__main__.TestAutoWrap) ... skip: Test Requires CUDA (0.002s)
2022-05-18T03:42:31.3129994Z   test_auto_wrap_smoke_test_fsdp_init_mode_FSDPInitMode_CUDA_BEFORE_cpu_offload_CPUOffload(offload_params=True)_use_device_id_True (__main__.TestAutoWrap) ... skip: Test Requires CUDA (0.002s)
2022-05-18T03:42:31.3142411Z   test_auto_wrap_with_ignored_modules_wrap_method_WrapMethod_FSDP_CTOR (__main__.TestAutoWrap) ... skip: Requires at least 2 GPUs (0.001s)
2022-05-18T03:42:31.3154822Z   test_auto_wrap_with_ignored_modules_wrap_method_WrapMethod_WRAP_API (__main__.TestAutoWrap) ... skip: Requires at least 2 GPUs (0.001s)
2022-05-18T03:42:31.3162198Z   test_transformer_auto_wrap_policy (__main__.TestAutoWrap) ... skip: Requires at least 2 GPUs (0.001s)
2022-05-18T03:42:31.3169072Z   test_wrap_disabled_outside_context (__main__.TestAutoWrap) ... skip: Requires at least 2 GPUs (0.001s)
2022-05-18T03:42:31.3174718Z   test_wrap_override_defaults (__main__.TestAutoWrap) ... skip: Requires at least 2 GPUs (0.001s)
2022-05-18T03:42:31.3183042Z   test_wrap_wrap_method_WrapMethod_FSDP_CTOR (__main__.TestAutoWrap) ... skip: Requires at least 2 GPUs (0.001s)
2022-05-18T03:42:31.3191008Z   test_wrap_wrap_method_WrapMethod_WRAP_API (__main__.TestAutoWrap) ... skip: Requires at least 2 GPUs (0.001s)
2022-05-18T03:42:31.3201605Z   test_bn_always_wrapped_individually (__main__.TestFSDPWrap)
2022-05-18T03:42:31.5988278Z Ensures that by using _or_policy with _wrap_batchnorm_individually, even ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24623
2022-05-18T03:42:31.6009747Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24624
2022-05-18T03:42:31.6032187Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24625
2022-05-18T03:42:31.6056335Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 24626
2022-05-18T03:42:32.2210917Z dist init r=0, world=4
2022-05-18T03:42:32.2375671Z dist init r=3, world=4
2022-05-18T03:42:32.2511202Z dist init r=1, world=4
2022-05-18T03:42:32.2629590Z dist init r=2, world=4
2022-05-18T03:42:32.2786284Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:42:32.2886145Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:42:32.2922709Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:42:32.2923265Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:42:32.2923886Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:42:32.2924435Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:42:32.2989391Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:42:32.2990201Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:42:32.3095873Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:42:32.3096549Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:42:32.3097200Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:42:32.3097819Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:42:32.5085382Z skip: Need at least 2 CUDA devices (1.189s)
2022-05-18T03:42:32.5092633Z   test_error_already_wrapped_nested_False_fsdp_init_mode_FSDPInitMode_CUDA_AFTER (__main__.TestFSDPWrap)
2022-05-18T03:42:32.5129540Z Test that an error is raised if we attempt to wrap when submodules are ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24679
2022-05-18T03:42:32.5155294Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24680
2022-05-18T03:42:32.5178196Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24681
2022-05-18T03:42:32.5202253Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 24682
2022-05-18T03:42:33.1293898Z dist init r=2, world=4
2022-05-18T03:42:33.1377539Z dist init r=1, world=4
2022-05-18T03:42:33.1505435Z dist init r=3, world=4
2022-05-18T03:42:33.1643648Z dist init r=0, world=4
2022-05-18T03:42:33.1815410Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:42:33.2019010Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:42:33.2019888Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:42:33.2020521Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:42:33.2021107Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:42:33.2021719Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:42:33.2022408Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:42:33.2023346Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:42:33.2027640Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:42:33.2029046Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:42:33.2030019Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:42:33.2030970Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:42:33.4228872Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:42:33.4236257Z   test_error_already_wrapped_nested_False_fsdp_init_mode_FSDPInitMode_CUDA_BEFORE (__main__.TestFSDPWrap)
2022-05-18T03:42:33.4273124Z Test that an error is raised if we attempt to wrap when submodules are ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24735
2022-05-18T03:42:33.4299438Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24736
2022-05-18T03:42:33.4322880Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24737
2022-05-18T03:42:33.4346976Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 24738
2022-05-18T03:42:34.0092785Z dist init r=1, world=4
2022-05-18T03:42:34.0155046Z dist init r=0, world=4
2022-05-18T03:42:34.0168735Z dist init r=3, world=4
2022-05-18T03:42:34.0717073Z dist init r=2, world=4
2022-05-18T03:42:34.1006487Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:42:34.1107718Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:42:34.1109411Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:42:34.1109938Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:42:34.1110827Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:42:34.1111744Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:42:34.1112660Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:42:34.1113408Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:42:34.1115997Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:42:34.1117196Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:42:34.1117821Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:42:34.1118341Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:42:34.3373118Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:42:34.3380437Z   test_error_already_wrapped_nested_True_fsdp_init_mode_FSDPInitMode_CUDA_AFTER (__main__.TestFSDPWrap)
2022-05-18T03:42:34.3417124Z Test that an error is raised if we attempt to wrap when submodules are ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24791
2022-05-18T03:42:34.3443345Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24792
2022-05-18T03:42:34.3466884Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24793
2022-05-18T03:42:34.3490305Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 24794
2022-05-18T03:42:34.9489738Z dist init r=1, world=4
2022-05-18T03:42:34.9603770Z dist init r=2, world=4
2022-05-18T03:42:34.9727220Z dist init r=3, world=4
2022-05-18T03:42:34.9850912Z dist init r=0, world=4
2022-05-18T03:42:35.0035846Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:42:35.0216307Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:42:35.0318119Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:42:35.0318838Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:42:35.0319474Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:42:35.0319991Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:42:35.0320515Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:42:35.0339939Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:42:35.0425603Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:42:35.0426127Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:42:35.0426658Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:42:35.0427196Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:42:35.2517495Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:42:35.2524315Z   test_error_already_wrapped_nested_True_fsdp_init_mode_FSDPInitMode_CUDA_BEFORE (__main__.TestFSDPWrap)
2022-05-18T03:42:35.2561019Z Test that an error is raised if we attempt to wrap when submodules are ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24847
2022-05-18T03:42:35.2587279Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24848
2022-05-18T03:42:35.2609803Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24849
2022-05-18T03:42:35.2633545Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 24850
2022-05-18T03:42:35.8397014Z dist init r=1, world=4
2022-05-18T03:42:35.8397381Z dist init r=0, world=4
2022-05-18T03:42:35.8872066Z dist init r=3, world=4
2022-05-18T03:42:35.9059285Z dist init r=2, world=4
2022-05-18T03:42:35.9268840Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:42:35.9369654Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:42:35.9370617Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:42:35.9371291Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:42:35.9372456Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:42:35.9373314Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:42:35.9374093Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:42:35.9374854Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:42:35.9378945Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:42:35.9379601Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:42:35.9380279Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:42:35.9380739Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:42:36.1659994Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:42:36.1722932Z   test_main_wrap_api_cpu_offload_CPUOffload(offload_params=False)_backward_prefetch_BackwardPrefetch_BACKWARD_POST_fsdp_init_mode_FSDPInitMode_CUDA_AFTER (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24903
2022-05-18T03:42:36.1748601Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24904
2022-05-18T03:42:36.1771803Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24905
2022-05-18T03:42:36.1795814Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 24906
2022-05-18T03:42:36.7848119Z dist init r=3, world=4
2022-05-18T03:42:36.8133629Z dist init r=1, world=4
2022-05-18T03:42:36.8236426Z dist init r=2, world=4
2022-05-18T03:42:36.8251050Z dist init r=0, world=4
2022-05-18T03:42:36.8546436Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:42:36.8646822Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:42:36.8748629Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:42:36.8749260Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:42:36.8750146Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:42:36.8750680Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:42:36.8751192Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:42:36.8755943Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:42:36.8761346Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:42:36.8762240Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:42:36.8762748Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:42:36.8763398Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:42:37.0822833Z skip: Need at least 2 CUDA devices (0.916s)
2022-05-18T03:42:37.0882796Z   test_main_wrap_api_cpu_offload_CPUOffload(offload_params=False)_backward_prefetch_BackwardPrefetch_BACKWARD_POST_fsdp_init_mode_FSDPInitMode_CUDA_BEFORE (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24959
2022-05-18T03:42:37.0909179Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24960
2022-05-18T03:42:37.0931997Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24961
2022-05-18T03:42:37.0955961Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 24962
2022-05-18T03:42:37.6806791Z dist init r=2, world=4
2022-05-18T03:42:37.7328970Z dist init r=3, world=4
2022-05-18T03:42:37.7360898Z dist init r=0, world=4
2022-05-18T03:42:37.7402148Z dist init r=1, world=4
2022-05-18T03:42:37.7536568Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:42:37.7713617Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:42:37.7714208Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:42:37.7714814Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:42:37.7715217Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:42:37.7715711Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:42:37.7716248Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:42:37.7739254Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:42:37.7821619Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:42:37.7822066Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:42:37.7822582Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:42:37.7823281Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:42:37.9983322Z skip: Need at least 2 CUDA devices (0.916s)
2022-05-18T03:42:38.0043150Z   test_main_wrap_api_cpu_offload_CPUOffload(offload_params=False)_backward_prefetch_BackwardPrefetch_BACKWARD_PRE_fsdp_init_mode_FSDPInitMode_CUDA_AFTER (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25015
2022-05-18T03:42:38.0069687Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25016
2022-05-18T03:42:38.0093166Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25017
2022-05-18T03:42:38.0117478Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 25018
2022-05-18T03:42:38.6251979Z dist init r=0, world=4
2022-05-18T03:42:38.6371305Z dist init r=2, world=4
2022-05-18T03:42:38.6546941Z dist init r=3, world=4
2022-05-18T03:42:38.6564008Z dist init r=1, world=4
2022-05-18T03:42:38.6757393Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:42:38.6959608Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:42:38.7061322Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:42:38.7062169Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:42:38.7063006Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:42:38.7063557Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:42:38.7064079Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:42:38.7064577Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:42:38.7169724Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:42:38.7170435Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:42:38.7170957Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:42:38.7171502Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:42:38.9144430Z skip: Need at least 2 CUDA devices (0.916s)
2022-05-18T03:42:38.9204950Z   test_main_wrap_api_cpu_offload_CPUOffload(offload_params=False)_backward_prefetch_BackwardPrefetch_BACKWARD_PRE_fsdp_init_mode_FSDPInitMode_CUDA_BEFORE (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25071
2022-05-18T03:42:38.9231044Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25072
2022-05-18T03:42:38.9255093Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25073
2022-05-18T03:42:38.9279266Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 25074
2022-05-18T03:42:39.5199384Z dist init r=1, world=4
2022-05-18T03:42:39.5750841Z dist init r=2, world=4
2022-05-18T03:42:39.5869895Z dist init r=0, world=4
2022-05-18T03:42:39.6142444Z dist init r=3, world=4
2022-05-18T03:42:39.6250210Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:42:39.6315402Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:42:39.6316250Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:42:39.6317088Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:42:39.6318400Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:42:39.6319370Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:42:39.6320420Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:42:39.6352733Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:42:39.6422590Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:42:39.6423229Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:42:39.6423729Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:42:39.6424267Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:42:39.8306138Z skip: Need at least 2 CUDA devices (0.916s)
2022-05-18T03:42:39.8366323Z   test_main_wrap_api_cpu_offload_CPUOffload(offload_params=True)_backward_prefetch_BackwardPrefetch_BACKWARD_POST_fsdp_init_mode_FSDPInitMode_CUDA_AFTER (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25127
2022-05-18T03:42:39.8392493Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25128
2022-05-18T03:42:39.8416263Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25129
2022-05-18T03:42:39.8439777Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 25130
2022-05-18T03:42:40.4668725Z dist init r=0, world=4
2022-05-18T03:42:40.4768375Z dist init r=2, world=4
2022-05-18T03:42:40.4816590Z dist init r=1, world=4
2022-05-18T03:42:40.4858448Z dist init r=3, world=4
2022-05-18T03:42:40.5079231Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:42:40.5180289Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:42:40.5283028Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:42:40.5283881Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:42:40.5285081Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:42:40.5285728Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:42:40.5286263Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:42:40.5286847Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:42:40.5289455Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:42:40.5289915Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:42:40.5290571Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:42:40.5292840Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:42:40.7467327Z skip: Need at least 2 CUDA devices (0.916s)
2022-05-18T03:42:40.7529434Z   test_main_wrap_api_cpu_offload_CPUOffload(offload_params=True)_backward_prefetch_BackwardPrefetch_BACKWARD_POST_fsdp_init_mode_FSDPInitMode_CUDA_BEFORE (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25183
2022-05-18T03:42:40.7555900Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25184
2022-05-18T03:42:40.7579232Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25185
2022-05-18T03:42:40.7603571Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 25186
2022-05-18T03:42:41.3820567Z dist init r=1, world=4
2022-05-18T03:42:41.3908021Z dist init r=0, world=4
2022-05-18T03:42:41.3953413Z dist init r=2, world=4
2022-05-18T03:42:41.4074969Z dist init r=3, world=4
2022-05-18T03:42:41.4183784Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:42:41.4386922Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:42:41.4488906Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:42:41.4489595Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:42:41.4490671Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:42:41.4491241Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:42:41.4491766Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:42:41.4589269Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:42:41.4596558Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:42:41.4597213Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:42:41.4597764Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:42:41.4598305Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:42:41.6630708Z skip: Need at least 2 CUDA devices (0.916s)
2022-05-18T03:42:41.6690331Z   test_main_wrap_api_cpu_offload_CPUOffload(offload_params=True)_backward_prefetch_BackwardPrefetch_BACKWARD_PRE_fsdp_init_mode_FSDPInitMode_CUDA_AFTER (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25239
2022-05-18T03:42:41.6716506Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25240
2022-05-18T03:42:41.6739881Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25241
2022-05-18T03:42:41.6763768Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 25242
2022-05-18T03:42:42.2679448Z dist init r=3, world=4
2022-05-18T03:42:42.2731546Z dist init r=2, world=4
2022-05-18T03:42:42.3001174Z dist init r=1, world=4
2022-05-18T03:42:42.3349749Z dist init r=0, world=4
2022-05-18T03:42:42.3644939Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:42:42.3746016Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:42:42.3848152Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:42:42.3849282Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:42:42.3850047Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:42:42.3850559Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:42:42.3851068Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:42:42.3851583Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:42:42.3909049Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:42:42.3909749Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:42:42.3910384Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:42:42.3911009Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:42:42.5791461Z skip: Need at least 2 CUDA devices (0.916s)
2022-05-18T03:42:42.5851946Z   test_main_wrap_api_cpu_offload_CPUOffload(offload_params=True)_backward_prefetch_BackwardPrefetch_BACKWARD_PRE_fsdp_init_mode_FSDPInitMode_CUDA_BEFORE (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25295
2022-05-18T03:42:42.5878538Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25296
2022-05-18T03:42:42.5902337Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25297
2022-05-18T03:42:42.5926185Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 25298
2022-05-18T03:42:43.2011401Z dist init r=1, world=4
2022-05-18T03:42:43.2172829Z dist init r=2, world=4
2022-05-18T03:42:43.2179219Z dist init r=0, world=4
2022-05-18T03:42:43.2336327Z dist init r=3, world=4
2022-05-18T03:42:43.2444422Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:42:43.2586118Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:42:43.2687560Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:42:43.2687973Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:42:43.2688611Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:42:43.2689130Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:42:43.2689653Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:42:43.2749573Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:42:43.2796422Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:42:43.2797204Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:42:43.2797753Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:42:43.2798592Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:42:43.4953330Z skip: Need at least 2 CUDA devices (0.916s)
2022-05-18T03:42:43.4998575Z   test_wrap_batchnorm_individually_use_or_policy_False (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25351
2022-05-18T03:42:43.5025640Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25352
2022-05-18T03:42:43.5050090Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25353
2022-05-18T03:42:43.5074555Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 25354
2022-05-18T03:42:44.1260956Z dist init r=1, world=4
2022-05-18T03:42:44.1347391Z dist init r=0, world=4
2022-05-18T03:42:44.1431254Z dist init r=2, world=4
2022-05-18T03:42:44.1598084Z dist init r=3, world=4
2022-05-18T03:42:44.1705740Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:42:44.1807360Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:42:44.1908590Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:42:44.1909147Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:42:44.1910075Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:42:44.1910920Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:42:44.1911655Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:42:44.1912183Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:42:44.2015897Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:42:44.2016446Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:42:44.2016913Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:42:44.2017268Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:42:44.4101420Z skip: Need at least 2 CUDA devices (0.915s)
2022-05-18T03:42:44.4146419Z   test_wrap_batchnorm_individually_use_or_policy_True (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25407
2022-05-18T03:42:44.4172014Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25408
2022-05-18T03:42:44.4196600Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25409
2022-05-18T03:42:44.4220682Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 25410
2022-05-18T03:42:45.0240585Z dist init r=0, world=4
2022-05-18T03:42:45.0684595Z dist init r=2, world=4
2022-05-18T03:42:45.0751533Z dist init r=1, world=4
2022-05-18T03:42:45.1163190Z dist init r=3, world=4
2022-05-18T03:42:45.1270765Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:42:45.1297964Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:42:45.1298688Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:42:45.1299360Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:42:45.1300111Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:42:45.1300640Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:42:45.1301161Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:42:45.1373282Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:42:45.1406393Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:42:45.1407083Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:42:45.1407561Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:42:45.1408092Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:42:45.3247078Z skip: Need at least 2 CUDA devices (0.914s)
2022-05-18T03:42:45.3247262Z 
2022-05-18T03:42:45.3247744Z ----------------------------------------------------------------------
2022-05-18T03:42:45.3248207Z Ran 38 tests in 14.031s
2022-05-18T03:42:45.3248376Z 
2022-05-18T03:42:45.3248452Z OK (skipped=38)
2022-05-18T03:42:45.3248560Z 
2022-05-18T03:42:45.3248645Z Generating XML reports...
2022-05-18T03:42:45.3303065Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_wrap/TEST-TestAutoWrap-20220518034231.xml
2022-05-18T03:42:45.3319393Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_wrap/TEST-TestFSDPWrap-20220518034231.xml
2022-05-18T03:42:45.5433368Z Running distributed/nn/jit/test_instantiator ... [2022-05-18 03:42:45.542954]
2022-05-18T03:42:45.5434299Z Executing ['/opt/conda/bin/python', 'distributed/nn/jit/test_instantiator.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-05-18 03:42:45.543030]
2022-05-18T03:42:46.0978930Z Test results will be stored in test-reports/python-unittest/distributed.nn.jit.test_instantiator
2022-05-18T03:42:46.0988691Z 
2022-05-18T03:42:46.0988833Z Running tests...
2022-05-18T03:42:46.0989418Z ----------------------------------------------------------------------
2022-05-18T03:42:46.3736779Z   test_get_arg_return_types_from_interface (__main__.TestInstantiator) ... ok (0.275s)
2022-05-18T03:42:46.3753962Z   test_instantiate_non_scripted_remote_module_template (__main__.TestInstantiator) ... ok (0.002s)
2022-05-18T03:42:46.3868964Z   test_instantiate_scripted_remote_module_template (__main__.TestInstantiator) ... ok (0.011s)
2022-05-18T03:42:46.3869352Z 
2022-05-18T03:42:46.3869820Z ----------------------------------------------------------------------
2022-05-18T03:42:46.3870280Z Ran 3 tests in 0.288s
2022-05-18T03:42:46.3870692Z 
2022-05-18T03:42:46.3870797Z OK
2022-05-18T03:42:46.3870943Z 
2022-05-18T03:42:46.3871077Z Generating XML reports...
2022-05-18T03:42:46.3895190Z Generated XML report: test-reports/python-unittest/distributed.nn.jit.test_instantiator/TEST-TestInstantiator-20220518034246.xml
2022-05-18T03:42:46.5552110Z Running distributed/optim/test_zero_redundancy_optimizer ... [2022-05-18 03:42:46.554754]
2022-05-18T03:42:46.5552696Z Executing ['/opt/conda/bin/python', 'distributed/optim/test_zero_redundancy_optimizer.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-05-18 03:42:46.554834]
2022-05-18T03:42:47.2982058Z Test results will be stored in test-reports/python-unittest/distributed.optim.test_zero_redundancy_optimizer
2022-05-18T03:42:47.2997883Z 
2022-05-18T03:42:47.2998028Z Running tests...
2022-05-18T03:42:47.2998618Z ----------------------------------------------------------------------
2022-05-18T03:42:47.3015462Z   test_add_param_group (__main__.TestZeroRedundancyOptimizerDistributed)
2022-05-18T03:42:47.5878027Z Check that ZeroRedundancyOptimizer properly handles adding a new ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/67287 for allplatform(s) . If you're seeing this on your local machine and would like to enable this test, please make sure IN_CI is not set and you are not using the flag --import-disabled-tests. (0.288s)
2022-05-18T03:42:47.5890477Z   test_collect_shards (__main__.TestZeroRedundancyOptimizerDistributed)
2022-05-18T03:42:47.5965405Z Check the state consolidation mechanism and the state dict exposed ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25484
2022-05-18T03:42:47.5989137Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25485
2022-05-18T03:42:48.3948275Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:42:48.4016067Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:42:48.6016846Z skip: CUDA is not available. (1.014s)
2022-05-18T03:42:48.6029747Z   test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_False_gradient_as_bucket_view_False_static_graph_False_shard_buckets_False (__main__.TestZeroRedundancyOptimizerDistributed)
2022-05-18T03:42:48.6030668Z Check that overlapping DDP with ZeRO using the given method determined ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-05-18T03:42:48.6041880Z   test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_False_gradient_as_bucket_view_False_static_graph_False_shard_buckets_True (__main__.TestZeroRedundancyOptimizerDistributed)
2022-05-18T03:42:48.6042412Z Check that overlapping DDP with ZeRO using the given method determined ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-05-18T03:42:48.6052287Z   test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_False_gradient_as_bucket_view_False_static_graph_True_shard_buckets_False (__main__.TestZeroRedundancyOptimizerDistributed)
2022-05-18T03:42:48.6052792Z Check that overlapping DDP with ZeRO using the given method determined ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-05-18T03:42:48.6062470Z   test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_False_gradient_as_bucket_view_False_static_graph_True_shard_buckets_True (__main__.TestZeroRedundancyOptimizerDistributed)
2022-05-18T03:42:48.6063094Z Check that overlapping DDP with ZeRO using the given method determined ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-05-18T03:42:48.6073171Z   test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_False_gradient_as_bucket_view_True_static_graph_False_shard_buckets_False (__main__.TestZeroRedundancyOptimizerDistributed)
2022-05-18T03:42:48.6073713Z Check that overlapping DDP with ZeRO using the given method determined ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-05-18T03:42:48.6083528Z   test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_False_gradient_as_bucket_view_True_static_graph_False_shard_buckets_True (__main__.TestZeroRedundancyOptimizerDistributed)
2022-05-18T03:42:48.6084324Z Check that overlapping DDP with ZeRO using the given method determined ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-05-18T03:42:48.6094537Z   test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_False_gradient_as_bucket_view_True_static_graph_True_shard_buckets_False (__main__.TestZeroRedundancyOptimizerDistributed)
2022-05-18T03:42:48.6095052Z Check that overlapping DDP with ZeRO using the given method determined ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-05-18T03:42:48.6104611Z   test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_False_gradient_as_bucket_view_True_static_graph_True_shard_buckets_True (__main__.TestZeroRedundancyOptimizerDistributed)
2022-05-18T03:42:48.6105196Z Check that overlapping DDP with ZeRO using the given method determined ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-05-18T03:42:48.6115342Z   test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_True_gradient_as_bucket_view_False_static_graph_False_shard_buckets_False (__main__.TestZeroRedundancyOptimizerDistributed)
2022-05-18T03:42:48.6115902Z Check that overlapping DDP with ZeRO using the given method determined ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-05-18T03:42:48.6125881Z   test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_True_gradient_as_bucket_view_False_static_graph_False_shard_buckets_True (__main__.TestZeroRedundancyOptimizerDistributed)
2022-05-18T03:42:48.6126476Z Check that overlapping DDP with ZeRO using the given method determined ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-05-18T03:42:48.6136946Z   test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_True_gradient_as_bucket_view_False_static_graph_True_shard_buckets_False (__main__.TestZeroRedundancyOptimizerDistributed)
2022-05-18T03:42:48.6137481Z Check that overlapping DDP with ZeRO using the given method determined ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-05-18T03:42:48.6147776Z   test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_True_gradient_as_bucket_view_False_static_graph_True_shard_buckets_True (__main__.TestZeroRedundancyOptimizerDistributed)
2022-05-18T03:42:48.6148292Z Check that overlapping DDP with ZeRO using the given method determined ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-05-18T03:42:48.6158405Z   test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_True_gradient_as_bucket_view_True_static_graph_False_shard_buckets_False (__main__.TestZeroRedundancyOptimizerDistributed)
2022-05-18T03:42:48.6158966Z Check that overlapping DDP with ZeRO using the given method determined ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-05-18T03:42:48.6169400Z   test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_True_gradient_as_bucket_view_True_static_graph_False_shard_buckets_True (__main__.TestZeroRedundancyOptimizerDistributed)
2022-05-18T03:42:48.6169931Z Check that overlapping DDP with ZeRO using the given method determined ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-05-18T03:42:48.6179850Z   test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_True_gradient_as_bucket_view_True_static_graph_True_shard_buckets_False (__main__.TestZeroRedundancyOptimizerDistributed)
2022-05-18T03:42:48.6180368Z Check that overlapping DDP with ZeRO using the given method determined ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-05-18T03:42:48.6190357Z   test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_True_gradient_as_bucket_view_True_static_graph_True_shard_buckets_True (__main__.TestZeroRedundancyOptimizerDistributed)
2022-05-18T03:42:48.6190864Z Check that overlapping DDP with ZeRO using the given method determined ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-05-18T03:42:48.6220882Z   test_local_optimizer_parity_optimizer_class_str_AdamW_maximize_False (__main__.TestZeroRedundancyOptimizerDistributed)
2022-05-18T03:42:48.6257261Z When combined with DDP, check that a local optimizer gives the same ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25506
2022-05-18T03:42:48.6284784Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25507
2022-05-18T03:42:49.3783545Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:42:49.3866542Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:42:49.5307965Z skip: CUDA is not available. (0.911s)
2022-05-18T03:42:49.5339569Z   test_local_optimizer_parity_optimizer_class_str_AdamW_maximize_True (__main__.TestZeroRedundancyOptimizerDistributed)
2022-05-18T03:42:49.5375667Z When combined with DDP, check that a local optimizer gives the same ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25528
2022-05-18T03:42:49.5401812Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25529
2022-05-18T03:42:50.2861012Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:42:50.2966144Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:42:50.4424344Z skip: CUDA is not available. (0.911s)
2022-05-18T03:42:50.4455798Z   test_local_optimizer_parity_optimizer_class_str_Adam_maximize_False (__main__.TestZeroRedundancyOptimizerDistributed)
2022-05-18T03:42:50.4495076Z When combined with DDP, check that a local optimizer gives the same ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25550
2022-05-18T03:42:50.4522762Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25551
2022-05-18T03:42:51.2170246Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:42:51.2242275Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:42:51.3548081Z skip: CUDA is not available. (0.912s)
2022-05-18T03:42:51.3579468Z   test_local_optimizer_parity_optimizer_class_str_Adam_maximize_True (__main__.TestZeroRedundancyOptimizerDistributed)
2022-05-18T03:42:51.3615920Z When combined with DDP, check that a local optimizer gives the same ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25572
2022-05-18T03:42:51.3643542Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25573
2022-05-18T03:42:52.1134440Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:42:52.1157902Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:42:52.2668705Z skip: CUDA is not available. (0.912s)
2022-05-18T03:42:52.2700112Z   test_local_optimizer_parity_optimizer_class_str_SGD_maximize_False (__main__.TestZeroRedundancyOptimizerDistributed)
2022-05-18T03:42:52.2736992Z When combined with DDP, check that a local optimizer gives the same ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25594
2022-05-18T03:42:52.2763693Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25595
2022-05-18T03:42:53.0253589Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:42:53.0312826Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:42:53.1788685Z skip: CUDA is not available. (0.912s)
2022-05-18T03:42:53.1819188Z   test_local_optimizer_parity_optimizer_class_str_SGD_maximize_True (__main__.TestZeroRedundancyOptimizerDistributed)
2022-05-18T03:42:53.1856125Z When combined with DDP, check that a local optimizer gives the same ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25616
2022-05-18T03:42:53.1883193Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25617
2022-05-18T03:42:53.9654240Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:42:53.9667202Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:42:54.0907935Z skip: CUDA is not available. (0.912s)
2022-05-18T03:42:54.0918402Z   test_lr_scheduler (__main__.TestZeroRedundancyOptimizerDistributed)
2022-05-18T03:42:54.0956006Z Check that a normal PyTorch ``lr_scheduler`` is usable with ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25638
2022-05-18T03:42:54.0983329Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25639
2022-05-18T03:42:54.8591249Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:42:54.8863158Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:42:55.0006639Z skip: CUDA is not available. (0.910s)
2022-05-18T03:42:55.0028120Z   test_multiple_param_groups (__main__.TestZeroRedundancyOptimizerDistributed)
2022-05-18T03:42:55.0064410Z Check parity between constructing ZeRO with multiple parameter groups ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25660
2022-05-18T03:42:55.0090886Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25661
2022-05-18T03:42:55.7768453Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:42:55.7775820Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:42:55.9115142Z skip: CUDA is not available. (0.911s)
2022-05-18T03:42:55.9139029Z   test_nondefault_process_group (__main__.TestZeroRedundancyOptimizerDistributed)
2022-05-18T03:42:55.9175653Z Check that ZeroRedundancyOptimizer works with a non-default process ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25682
2022-05-18T03:42:55.9201723Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25683
2022-05-18T03:42:56.6710300Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:42:56.6711088Z INFO:torch.testing._internal.common_distributed:Skipping `test_nondefault_process_group()` since world size of 2 is less than 4
2022-05-18T03:42:56.7074518Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:42:56.7075252Z INFO:torch.testing._internal.common_distributed:Skipping `test_nondefault_process_group()` since world size of 2 is less than 4
2022-05-18T03:42:56.8225085Z ok (0.911s)
2022-05-18T03:42:56.8233077Z   test_sharding (__main__.TestZeroRedundancyOptimizerDistributed)
2022-05-18T03:42:56.8236714Z Check ZeroRedundancyOptimizer's parameter sharding at construction ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/67295 for allplatform(s) . If you're seeing this on your local machine and would like to enable this test, please make sure IN_CI is not set and you are not using the flag --import-disabled-tests. (0.001s)
2022-05-18T03:42:56.8249210Z   test_step (__main__.TestZeroRedundancyOptimizerDistributed)
2022-05-18T03:42:56.8284619Z Check that ZeroRedundancyOptimizer properly exposes the ``step()`` ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25704
2022-05-18T03:42:56.8310930Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25705
2022-05-18T03:42:57.5754354Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:42:57.5851875Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:42:57.7333709Z skip: CUDA is not available. (0.910s)
2022-05-18T03:42:57.7351081Z   test_step_with_closure (__main__.TestZeroRedundancyOptimizerDistributed)
2022-05-18T03:42:57.7386430Z Check that ZeroRedundancyOptimizer properly exposes the ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25726
2022-05-18T03:42:57.7413013Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25727
2022-05-18T03:42:58.4802693Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:42:58.4878863Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:42:58.6435017Z skip: CUDA is not available. (0.910s)
2022-05-18T03:42:58.6439491Z   test_zero_join_cpu (__main__.TestZeroRedundancyOptimizerDistributed)
2022-05-18T03:42:58.6476221Z Check that the ZeRO join hook allows training with uneven inputs ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25748
2022-05-18T03:42:58.6501955Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25749
2022-05-18T03:42:59.4148132Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:42:59.4407060Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:42:59.4518072Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:42:59.4518510Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:42:59.4519133Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:42:59.4519667Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T03:42:59.4591788Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppapxiw1k
2022-05-18T03:42:59.4593485Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppapxiw1k/_remote_module_non_scriptable.py
2022-05-18T03:42:59.4689132Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuetscqym
2022-05-18T03:42:59.4691271Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuetscqym/_remote_module_non_scriptable.py
2022-05-18T03:42:59.4837321Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T03:42:59.4837695Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T03:42:59.5124829Z /opt/conda/lib/python3.7/tempfile.py:798: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmpuetscqym'>
2022-05-18T03:42:59.5125204Z   _warnings.warn(warn_message, ResourceWarning)
2022-05-18T03:42:59.5125713Z /opt/conda/lib/python3.7/tempfile.py:798: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmppapxiw1k'>
2022-05-18T03:42:59.5126041Z   _warnings.warn(warn_message, ResourceWarning)
2022-05-18T03:42:59.6526470Z ok (1.009s)
2022-05-18T03:42:59.6530629Z   test_zero_join_gpu (__main__.TestZeroRedundancyOptimizerDistributed)
2022-05-18T03:42:59.6531497Z Check that the ZeRO join hook allows training with uneven inputs ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-05-18T03:42:59.6535437Z   test_zero_model_parallel_parameters_as_bucket_view_False (__main__.TestZeroRedundancyOptimizerDistributed)
2022-05-18T03:42:59.6571311Z Check that ZeRO works with model parallelism where the model's ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25776
2022-05-18T03:42:59.6596876Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25777
2022-05-18T03:43:00.3897267Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:43:00.3916512Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:43:00.5620950Z skip: Need at least 4 CUDA devices (0.909s)
2022-05-18T03:43:00.5626717Z   test_zero_model_parallel_parameters_as_bucket_view_True (__main__.TestZeroRedundancyOptimizerDistributed)
2022-05-18T03:43:00.5664150Z Check that ZeRO works with model parallelism where the model's ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25798
2022-05-18T03:43:00.5691466Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25799
2022-05-18T03:43:01.3084945Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:43:01.3225455Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:43:01.4714344Z skip: Need at least 4 CUDA devices (0.909s)
2022-05-18T03:43:01.4731176Z   test_constructor (__main__.TestZeroRedundancyOptimizerSingleRank)
2022-05-18T03:43:01.4769115Z Check the robustness of the ZeroRedundancyOptimizer constructor by ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25820
2022-05-18T03:43:02.1715438Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:43:02.1721615Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:43:02.1722562Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-05-18T03:43:02.3792107Z ok (0.908s)
2022-05-18T03:43:02.3803228Z   test_lr_scheduler (__main__.TestZeroRedundancyOptimizerSingleRank)
2022-05-18T03:43:02.3840559Z Check that a normal PyTorch ``lr_scheduler`` is usable with ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25834
2022-05-18T03:43:03.1203365Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:43:03.1209768Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:43:03.1210527Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-05-18T03:43:03.2861961Z ok (0.907s)
2022-05-18T03:43:03.2870432Z   test_same_dense_param_type (__main__.TestZeroRedundancyOptimizerSingleRank)
2022-05-18T03:43:03.2907079Z Check that ZeroRedundancyOptimizer raises an exception if the input ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25848
2022-05-18T03:43:03.9839668Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:43:03.9845967Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:43:03.9846970Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-05-18T03:43:04.0929932Z ok (0.807s)
2022-05-18T03:43:04.0953207Z   test_state_dict (__main__.TestZeroRedundancyOptimizerSingleRank)
2022-05-18T03:43:04.0989319Z Check that ZeroRedundancyOptimizer exposes the expected state dict ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25862
2022-05-18T03:43:04.7932692Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:43:04.7938625Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:43:04.7939394Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-05-18T03:43:05.0011826Z ok (0.908s)
2022-05-18T03:43:05.0021396Z   test_step_with_extra_inner_key (__main__.TestZeroRedundancyOptimizerSingleRank)
2022-05-18T03:43:05.0058886Z Check that ZeroRedundancyOptimizer wrapping an optimizer that adds ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25876
2022-05-18T03:43:05.7394241Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:43:05.7400345Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:43:05.7401032Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-05-18T03:43:05.9080047Z ok (0.907s)
2022-05-18T03:43:05.9089801Z   test_step_with_kwargs (__main__.TestZeroRedundancyOptimizerSingleRank)
2022-05-18T03:43:05.9126310Z Check that the ``step(**kwargs)`` interface is properly exposed. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25890
2022-05-18T03:43:06.6432341Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:43:06.6438557Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:43:06.6439251Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-05-18T03:43:06.8148072Z ok (0.907s)
2022-05-18T03:43:06.8155747Z   test_step_without_closure (__main__.TestZeroRedundancyOptimizerSingleRank)
2022-05-18T03:43:06.8191885Z Check that the ``step()`` method (without closure) is handled as ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25904
2022-05-18T03:43:07.5567184Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:43:07.5573383Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:43:07.5574186Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-05-18T03:43:07.7213698Z ok (0.906s)
2022-05-18T03:43:07.7223144Z   test_zero_grad (__main__.TestZeroRedundancyOptimizerSingleRank)
2022-05-18T03:43:07.7259653Z Check that the ``zero_grad`` method is properly handled. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25918
2022-05-18T03:43:08.4625817Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:43:08.4632592Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:43:08.4633544Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-05-18T03:43:08.6282601Z ok (0.907s)
2022-05-18T03:43:08.6282839Z 
2022-05-18T03:43:08.6283297Z ----------------------------------------------------------------------
2022-05-18T03:43:08.6283678Z Ran 42 tests in 21.328s
2022-05-18T03:43:08.6283875Z 
2022-05-18T03:43:08.6284002Z OK (skipped=32)
2022-05-18T03:43:08.6284164Z 
2022-05-18T03:43:08.6284298Z Generating XML reports...
2022-05-18T03:43:08.6352925Z Generated XML report: test-reports/python-unittest/distributed.optim.test_zero_redundancy_optimizer/TEST-TestZeroRedundancyOptimizerDistributed-20220518034247.xml
2022-05-18T03:43:08.6362273Z Generated XML report: test-reports/python-unittest/distributed.optim.test_zero_redundancy_optimizer/TEST-TestZeroRedundancyOptimizerSingleRank-20220518034247.xml
2022-05-18T03:43:08.8214607Z Running distributed/pipeline/sync/skip/test_api ... [2022-05-18 03:43:08.821054]
2022-05-18T03:43:08.8215443Z Executing ['/opt/conda/bin/python', '-m', 'pytest', 'distributed/pipeline/sync/skip/test_api.py', '-v'] ... [2022-05-18 03:43:08.821138]
2022-05-18T03:43:09.6223326Z ============================= test session starts ==============================
2022-05-18T03:43:09.6223779Z platform linux -- Python 3.7.13, pytest-7.1.2, pluggy-1.0.0 -- /opt/conda/bin/python
2022-05-18T03:43:09.6237483Z cachedir: .pytest_cache
2022-05-18T03:43:09.6238077Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2022-05-18T03:43:09.6238419Z torch: 1.12.0a0+git3b23752
2022-05-18T03:43:09.6238663Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2022-05-18T03:43:09.6238934Z plugins: hypothesis-4.53.2
2022-05-18T03:43:09.6345096Z collecting ... 
2022-05-18T03:43:09.6345407Z collected 3 items                                                              
2022-05-18T03:43:09.6348902Z 
2022-05-18T03:43:09.6372862Z distributed/pipeline/sync/skip/test_api.py::test_namespace_difference PASSED [ 33%]
2022-05-18T03:43:09.6384463Z distributed/pipeline/sync/skip/test_api.py::test_namespace_copy PASSED   [ 66%]
2022-05-18T03:43:09.6406039Z distributed/pipeline/sync/skip/test_api.py::test_skippable_repr PASSED   [100%]
2022-05-18T03:43:09.6407379Z 
2022-05-18T03:43:09.6407683Z ============================== 3 passed in 0.02s ===============================
2022-05-18T03:43:09.7623695Z Running distributed/pipeline/sync/skip/test_gpipe ... [2022-05-18 03:43:09.762036]
2022-05-18T03:43:09.7624241Z Executing ['/opt/conda/bin/python', '-m', 'pytest', 'distributed/pipeline/sync/skip/test_gpipe.py', '-v'] ... [2022-05-18 03:43:09.762118]
2022-05-18T03:43:10.5656813Z ============================= test session starts ==============================
2022-05-18T03:43:10.5657244Z platform linux -- Python 3.7.13, pytest-7.1.2, pluggy-1.0.0 -- /opt/conda/bin/python
2022-05-18T03:43:10.5670779Z cachedir: .pytest_cache
2022-05-18T03:43:10.5671451Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2022-05-18T03:43:10.5671895Z torch: 1.12.0a0+git3b23752
2022-05-18T03:43:10.5672174Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2022-05-18T03:43:10.5672462Z plugins: hypothesis-4.53.2
2022-05-18T03:43:10.5815644Z collecting ... 
2022-05-18T03:43:10.5816005Z collected 13 items                                                             
2022-05-18T03:43:10.5819109Z 
2022-05-18T03:43:10.5833677Z distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[never-3] SKIPPED [  7%]
2022-05-18T03:43:10.5841114Z distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[never-1:2] SKIPPED [ 15%]
2022-05-18T03:43:10.5848162Z distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[never-2:1] SKIPPED [ 23%]
2022-05-18T03:43:10.5855537Z distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[never-1:1:1] SKIPPED [ 30%]
2022-05-18T03:43:10.5863547Z distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[always-3] SKIPPED [ 38%]
2022-05-18T03:43:10.5870882Z distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[always-1:2] SKIPPED [ 46%]
2022-05-18T03:43:10.5878299Z distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[always-2:1] SKIPPED [ 53%]
2022-05-18T03:43:10.5885694Z distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[always-1:1:1] SKIPPED [ 61%]
2022-05-18T03:43:10.5893149Z distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[except_last-3] SKIPPED [ 69%]
2022-05-18T03:43:10.5901624Z distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[except_last-1:2] SKIPPED [ 76%]
2022-05-18T03:43:10.5909326Z distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[except_last-2:1] SKIPPED [ 84%]
2022-05-18T03:43:10.5916609Z distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[except_last-1:1:1] SKIPPED [ 92%]
2022-05-18T03:43:10.6357139Z distributed/pipeline/sync/skip/test_gpipe.py::test_none_skip PASSED      [100%]
2022-05-18T03:43:10.6362510Z 
2022-05-18T03:43:10.6362686Z =========================== short test summary info ============================
2022-05-18T03:43:10.6363185Z SKIPPED [12] distributed/pipeline/sync/skip/test_gpipe.py:19: cuda required
2022-05-18T03:43:10.6364194Z ======================== 1 passed, 12 skipped in 0.07s =========================
2022-05-18T03:43:10.7644477Z Running distributed/pipeline/sync/skip/test_inspect_skip_layout ... [2022-05-18 03:43:10.763999]
2022-05-18T03:43:10.7644997Z Executing ['/opt/conda/bin/python', '-m', 'pytest', 'distributed/pipeline/sync/skip/test_inspect_skip_layout.py', '-v'] ... [2022-05-18 03:43:10.764079]
2022-05-18T03:43:11.5589909Z ============================= test session starts ==============================
2022-05-18T03:43:11.5590615Z platform linux -- Python 3.7.13, pytest-7.1.2, pluggy-1.0.0 -- /opt/conda/bin/python
2022-05-18T03:43:11.5603982Z cachedir: .pytest_cache
2022-05-18T03:43:11.5604854Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2022-05-18T03:43:11.5605242Z torch: 1.12.0a0+git3b23752
2022-05-18T03:43:11.5605488Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2022-05-18T03:43:11.5605755Z plugins: hypothesis-4.53.2
2022-05-18T03:43:11.5726272Z collecting ... 
2022-05-18T03:43:11.5726612Z collected 6 items                                                              
2022-05-18T03:43:11.5729928Z 
2022-05-18T03:43:11.5755191Z distributed/pipeline/sync/skip/test_inspect_skip_layout.py::test_no_skippables PASSED [ 16%]
2022-05-18T03:43:11.5769610Z distributed/pipeline/sync/skip/test_inspect_skip_layout.py::test_inner_partition PASSED [ 33%]
2022-05-18T03:43:11.5782326Z distributed/pipeline/sync/skip/test_inspect_skip_layout.py::test_adjoining_partitions PASSED [ 50%]
2022-05-18T03:43:11.5795870Z distributed/pipeline/sync/skip/test_inspect_skip_layout.py::test_far_partitions PASSED [ 66%]
2022-05-18T03:43:11.5809926Z distributed/pipeline/sync/skip/test_inspect_skip_layout.py::test_pop_2_from_different_partitions PASSED [ 83%]
2022-05-18T03:43:11.5826641Z distributed/pipeline/sync/skip/test_inspect_skip_layout.py::test_namespace PASSED [100%]
2022-05-18T03:43:11.5828207Z 
2022-05-18T03:43:11.5828605Z ============================== 6 passed in 0.02s ===============================
2022-05-18T03:43:11.7032589Z Running distributed/pipeline/sync/skip/test_leak ... [2022-05-18 03:43:11.702932]
2022-05-18T03:43:11.7033099Z Executing ['/opt/conda/bin/python', '-m', 'pytest', 'distributed/pipeline/sync/skip/test_leak.py', '-v'] ... [2022-05-18 03:43:11.703012]
2022-05-18T03:43:12.5075745Z ============================= test session starts ==============================
2022-05-18T03:43:12.5076240Z platform linux -- Python 3.7.13, pytest-7.1.2, pluggy-1.0.0 -- /opt/conda/bin/python
2022-05-18T03:43:12.5090185Z cachedir: .pytest_cache
2022-05-18T03:43:12.5090991Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2022-05-18T03:43:12.5091401Z torch: 1.12.0a0+git3b23752
2022-05-18T03:43:12.5091648Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2022-05-18T03:43:12.5091934Z plugins: hypothesis-4.53.2
2022-05-18T03:43:12.5219179Z collecting ... 
2022-05-18T03:43:12.5219559Z collected 8 items                                                              
2022-05-18T03:43:12.5222818Z 
2022-05-18T03:43:12.5755734Z distributed/pipeline/sync/skip/test_leak.py::test_delete_portal_tensor[always-train] PASSED [ 12%]
2022-05-18T03:43:12.5922254Z distributed/pipeline/sync/skip/test_leak.py::test_delete_portal_tensor[always-eval] PASSED [ 25%]
2022-05-18T03:43:12.6121815Z distributed/pipeline/sync/skip/test_leak.py::test_delete_portal_tensor[except_last-train] PASSED [ 37%]
2022-05-18T03:43:12.6360143Z distributed/pipeline/sync/skip/test_leak.py::test_delete_portal_tensor[except_last-eval] PASSED [ 50%]
2022-05-18T03:43:12.6559524Z distributed/pipeline/sync/skip/test_leak.py::test_delete_portal_tensor[never-train] PASSED [ 62%]
2022-05-18T03:43:12.6760478Z distributed/pipeline/sync/skip/test_leak.py::test_delete_portal_tensor[never-eval] PASSED [ 75%]
2022-05-18T03:43:12.6959923Z distributed/pipeline/sync/skip/test_leak.py::test_no_portal_without_pipe[train] PASSED [ 87%]
2022-05-18T03:43:12.7100570Z distributed/pipeline/sync/skip/test_leak.py::test_no_portal_without_pipe[eval] PASSED [100%]
2022-05-18T03:43:12.7101469Z 
2022-05-18T03:43:12.7101821Z ============================== 8 passed in 0.20s ===============================
2022-05-18T03:43:12.8389957Z Running distributed/pipeline/sync/skip/test_portal ... [2022-05-18 03:43:12.838600]
2022-05-18T03:43:12.8390467Z Executing ['/opt/conda/bin/python', '-m', 'pytest', 'distributed/pipeline/sync/skip/test_portal.py', '-v'] ... [2022-05-18 03:43:12.838679]
2022-05-18T03:43:13.6442881Z ============================= test session starts ==============================
2022-05-18T03:43:13.6443313Z platform linux -- Python 3.7.13, pytest-7.1.2, pluggy-1.0.0 -- /opt/conda/bin/python
2022-05-18T03:43:13.6457270Z cachedir: .pytest_cache
2022-05-18T03:43:13.6458067Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2022-05-18T03:43:13.6458672Z torch: 1.12.0a0+git3b23752
2022-05-18T03:43:13.6459136Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2022-05-18T03:43:13.6459434Z plugins: hypothesis-4.53.2
2022-05-18T03:43:13.6684001Z collecting ... 
2022-05-18T03:43:13.6684343Z collected 10 items                                                             
2022-05-18T03:43:13.6687461Z 
2022-05-18T03:43:13.6698099Z distributed/pipeline/sync/skip/test_portal.py::test_copy_returns_on_next_device SKIPPED [ 10%]
2022-05-18T03:43:13.6730993Z distributed/pipeline/sync/skip/test_portal.py::test_blue_orange PASSED   [ 20%]
2022-05-18T03:43:13.6746270Z distributed/pipeline/sync/skip/test_portal.py::test_blue_orange_not_requires_grad PASSED [ 30%]
2022-05-18T03:43:13.6758511Z distributed/pipeline/sync/skip/test_portal.py::test_use_grad PASSED      [ 40%]
2022-05-18T03:43:13.6771619Z distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_0 PASSED [ 50%]
2022-05-18T03:43:13.6784596Z distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_1 PASSED [ 60%]
2022-05-18T03:43:13.6797559Z distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_2 PASSED [ 70%]
2022-05-18T03:43:13.6810381Z distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_3 PASSED [ 80%]
2022-05-18T03:43:13.6823481Z distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_4 PASSED [ 90%]
2022-05-18T03:43:13.6839154Z distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_3_plus_1 PASSED [100%]
2022-05-18T03:43:13.6840447Z 
2022-05-18T03:43:13.6840795Z =========================== short test summary info ============================
2022-05-18T03:43:13.6841339Z SKIPPED [1] distributed/pipeline/sync/skip/test_portal.py:17: cuda required
2022-05-18T03:43:13.6842467Z ========================= 9 passed, 1 skipped in 0.04s =========================
2022-05-18T03:43:13.8051666Z Running distributed/pipeline/sync/skip/test_stash_pop ... [2022-05-18 03:43:13.804814]
2022-05-18T03:43:13.8052186Z Executing ['/opt/conda/bin/python', '-m', 'pytest', 'distributed/pipeline/sync/skip/test_stash_pop.py', '-v'] ... [2022-05-18 03:43:13.804895]
2022-05-18T03:43:14.6043609Z ============================= test session starts ==============================
2022-05-18T03:43:14.6044066Z platform linux -- Python 3.7.13, pytest-7.1.2, pluggy-1.0.0 -- /opt/conda/bin/python
2022-05-18T03:43:14.6058131Z cachedir: .pytest_cache
2022-05-18T03:43:14.6059196Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2022-05-18T03:43:14.6059624Z torch: 1.12.0a0+git3b23752
2022-05-18T03:43:14.6060044Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2022-05-18T03:43:14.6060579Z plugins: hypothesis-4.53.2
2022-05-18T03:43:14.6179002Z collecting ... 
2022-05-18T03:43:14.6179523Z collected 7 items                                                              
2022-05-18T03:43:14.6182273Z 
2022-05-18T03:43:14.6215189Z distributed/pipeline/sync/skip/test_stash_pop.py::test_stash PASSED      [ 14%]
2022-05-18T03:43:14.6230289Z distributed/pipeline/sync/skip/test_stash_pop.py::test_pop PASSED        [ 28%]
2022-05-18T03:43:14.6245194Z distributed/pipeline/sync/skip/test_stash_pop.py::test_declare_but_not_use PASSED [ 42%]
2022-05-18T03:43:14.6258173Z distributed/pipeline/sync/skip/test_stash_pop.py::test_stash_not_declared PASSED [ 57%]
2022-05-18T03:43:14.6272759Z distributed/pipeline/sync/skip/test_stash_pop.py::test_pop_not_declared PASSED [ 71%]
2022-05-18T03:43:14.6285513Z distributed/pipeline/sync/skip/test_stash_pop.py::test_pop_not_stashed PASSED [ 85%]
2022-05-18T03:43:14.6301091Z distributed/pipeline/sync/skip/test_stash_pop.py::test_stash_none PASSED [100%]
2022-05-18T03:43:14.6302435Z 
2022-05-18T03:43:14.6302707Z ============================== 7 passed in 0.03s ===============================
2022-05-18T03:43:14.7573413Z Running distributed/pipeline/sync/skip/test_tracker ... [2022-05-18 03:43:14.756926]
2022-05-18T03:43:14.7573930Z Executing ['/opt/conda/bin/python', '-m', 'pytest', 'distributed/pipeline/sync/skip/test_tracker.py', '-v'] ... [2022-05-18 03:43:14.757011]
2022-05-18T03:43:15.5604625Z ============================= test session starts ==============================
2022-05-18T03:43:15.5605035Z platform linux -- Python 3.7.13, pytest-7.1.2, pluggy-1.0.0 -- /opt/conda/bin/python
2022-05-18T03:43:15.5619233Z cachedir: .pytest_cache
2022-05-18T03:43:15.5619887Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2022-05-18T03:43:15.5620279Z torch: 1.12.0a0+git3b23752
2022-05-18T03:43:15.5620545Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2022-05-18T03:43:15.5620899Z plugins: hypothesis-4.53.2
2022-05-18T03:43:15.5784071Z collecting ... 
2022-05-18T03:43:15.5784483Z collected 6 items                                                              
2022-05-18T03:43:15.5787365Z 
2022-05-18T03:43:15.5814239Z distributed/pipeline/sync/skip/test_tracker.py::test_default_skip_tracker PASSED [ 16%]
2022-05-18T03:43:15.5823312Z distributed/pipeline/sync/skip/test_tracker.py::test_default_skip_tracker_by_data_parallel SKIPPED [ 33%]
2022-05-18T03:43:15.5840273Z distributed/pipeline/sync/skip/test_tracker.py::test_reuse_portal PASSED [ 50%]
2022-05-18T03:43:15.5853221Z distributed/pipeline/sync/skip/test_tracker.py::test_no_copy_no_portal PASSED [ 66%]
2022-05-18T03:43:15.5866219Z distributed/pipeline/sync/skip/test_tracker.py::test_tensor_life_without_checkpointing PASSED [ 83%]
2022-05-18T03:43:15.5881762Z distributed/pipeline/sync/skip/test_tracker.py::test_tensor_life_with_checkpointing PASSED [100%]
2022-05-18T03:43:15.5883344Z 
2022-05-18T03:43:15.5883634Z =========================== short test summary info ============================
2022-05-18T03:43:15.5884112Z SKIPPED [1] distributed/pipeline/sync/skip/test_tracker.py:39: cuda required
2022-05-18T03:43:15.5885152Z ========================= 5 passed, 1 skipped in 0.03s =========================
2022-05-18T03:43:15.7119167Z Running distributed/pipeline/sync/skip/test_verify_skippables ... [2022-05-18 03:43:15.711555]
2022-05-18T03:43:15.7119734Z Executing ['/opt/conda/bin/python', '-m', 'pytest', 'distributed/pipeline/sync/skip/test_verify_skippables.py', '-v'] ... [2022-05-18 03:43:15.711634]
2022-05-18T03:43:16.5059539Z ============================= test session starts ==============================
2022-05-18T03:43:16.5059952Z platform linux -- Python 3.7.13, pytest-7.1.2, pluggy-1.0.0 -- /opt/conda/bin/python
2022-05-18T03:43:16.5074827Z cachedir: .pytest_cache
2022-05-18T03:43:16.5075666Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2022-05-18T03:43:16.5076011Z torch: 1.12.0a0+git3b23752
2022-05-18T03:43:16.5076255Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2022-05-18T03:43:16.5076528Z plugins: hypothesis-4.53.2
2022-05-18T03:43:16.5213129Z collecting ... 
2022-05-18T03:43:16.5213568Z collected 9 items                                                              
2022-05-18T03:43:16.5215813Z 
2022-05-18T03:43:16.5242844Z distributed/pipeline/sync/skip/test_verify_skippables.py::test_matching PASSED [ 11%]
2022-05-18T03:43:16.5256114Z distributed/pipeline/sync/skip/test_verify_skippables.py::test_stash_not_pop PASSED [ 22%]
2022-05-18T03:43:16.5269552Z distributed/pipeline/sync/skip/test_verify_skippables.py::test_pop_unknown PASSED [ 33%]
2022-05-18T03:43:16.5282983Z distributed/pipeline/sync/skip/test_verify_skippables.py::test_stash_again PASSED [ 44%]
2022-05-18T03:43:16.5296874Z distributed/pipeline/sync/skip/test_verify_skippables.py::test_pop_again PASSED [ 55%]
2022-05-18T03:43:16.5317008Z distributed/pipeline/sync/skip/test_verify_skippables.py::test_stash_pop_together_different_names PASSED [ 66%]
2022-05-18T03:43:16.5329006Z distributed/pipeline/sync/skip/test_verify_skippables.py::test_stash_pop_together_same_name PASSED [ 77%]
2022-05-18T03:43:16.5343267Z distributed/pipeline/sync/skip/test_verify_skippables.py::test_double_stash_pop PASSED [ 88%]
2022-05-18T03:43:16.5360813Z distributed/pipeline/sync/skip/test_verify_skippables.py::test_double_stash_pop_but_isolated PASSED [100%]
2022-05-18T03:43:16.5361685Z 
2022-05-18T03:43:16.5362194Z ============================== 9 passed in 0.03s ===============================
2022-05-18T03:43:16.6608183Z Running distributed/pipeline/sync/test_balance ... [2022-05-18 03:43:16.660437]
2022-05-18T03:43:16.6608722Z Executing ['/opt/conda/bin/python', '-m', 'pytest', 'distributed/pipeline/sync/test_balance.py', '-v'] ... [2022-05-18 03:43:16.660523]
2022-05-18T03:43:17.4658907Z ============================= test session starts ==============================
2022-05-18T03:43:17.4659320Z platform linux -- Python 3.7.13, pytest-7.1.2, pluggy-1.0.0 -- /opt/conda/bin/python
2022-05-18T03:43:17.4673859Z cachedir: .pytest_cache
2022-05-18T03:43:17.4674621Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2022-05-18T03:43:17.4675224Z torch: 1.12.0a0+git3b23752
2022-05-18T03:43:17.4675670Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2022-05-18T03:43:17.4676179Z plugins: hypothesis-4.53.2
2022-05-18T03:43:17.4880528Z collecting ... 
2022-05-18T03:43:17.4881113Z collected 15 items                                                             
2022-05-18T03:43:17.4883752Z 
2022-05-18T03:43:17.4907090Z distributed/pipeline/sync/test_balance.py::test_blockpartition PASSED    [  6%]
2022-05-18T03:43:17.4918449Z distributed/pipeline/sync/test_balance.py::test_blockpartition_zeros PASSED [ 13%]
2022-05-18T03:43:17.4929945Z distributed/pipeline/sync/test_balance.py::test_blockpartition_non_positive_partitions PASSED [ 20%]
2022-05-18T03:43:17.4940896Z distributed/pipeline/sync/test_balance.py::test_blockpartition_short_sequence PASSED [ 26%]
2022-05-18T03:43:17.4949357Z distributed/pipeline/sync/test_balance.py::test_balance_by_time[cpu] SKIPPED [ 33%]
2022-05-18T03:43:18.4991127Z distributed/pipeline/sync/test_balance.py::test_balance_by_time_loop_resets_input PASSED [ 40%]
2022-05-18T03:43:18.4999196Z distributed/pipeline/sync/test_balance.py::test_balance_by_size_latent SKIPPED [ 46%]
2022-05-18T03:43:18.5007130Z distributed/pipeline/sync/test_balance.py::test_balance_by_size_param SKIPPED [ 53%]
2022-05-18T03:43:18.5014268Z distributed/pipeline/sync/test_balance.py::test_balance_by_size_param_scale SKIPPED [ 60%]
2022-05-18T03:43:18.5038929Z distributed/pipeline/sync/test_balance.py::test_layerwise_sandbox[cpu] PASSED [ 66%]
2022-05-18T03:43:19.5068178Z distributed/pipeline/sync/test_balance.py::test_sandbox_during_profiling[cpu] PASSED [ 73%]
2022-05-18T03:43:20.5109316Z distributed/pipeline/sync/test_balance.py::test_not_training PASSED      [ 80%]
2022-05-18T03:43:21.5142604Z distributed/pipeline/sync/test_balance.py::test_balance_by_time_tuple PASSED [ 86%]
2022-05-18T03:43:21.5150489Z distributed/pipeline/sync/test_balance.py::test_balance_by_size_tuple SKIPPED [ 93%]
2022-05-18T03:43:21.5176271Z distributed/pipeline/sync/test_balance.py::test_already_has_grad PASSED  [100%]
2022-05-18T03:43:21.5179617Z 
2022-05-18T03:43:21.5179832Z =========================== short test summary info ============================
2022-05-18T03:43:21.5180348Z SKIPPED [1] distributed/pipeline/sync/test_balance.py:47: Flaky due to time.sleep()
2022-05-18T03:43:21.5180947Z SKIPPED [1] distributed/pipeline/sync/test_balance.py:77: cuda required
2022-05-18T03:43:21.5181273Z SKIPPED [1] distributed/pipeline/sync/test_balance.py:100: cuda required
2022-05-18T03:43:21.5181604Z SKIPPED [1] distributed/pipeline/sync/test_balance.py:113: cuda required
2022-05-18T03:43:21.5181940Z SKIPPED [1] distributed/pipeline/sync/test_balance.py:204: cuda required
2022-05-18T03:43:21.5182362Z ======================== 10 passed, 5 skipped in 4.05s =========================
2022-05-18T03:43:21.6462361Z Running distributed/pipeline/sync/test_bugs ... [2022-05-18 03:43:21.645910]
2022-05-18T03:43:21.6463005Z Executing ['/opt/conda/bin/python', '-m', 'pytest', 'distributed/pipeline/sync/test_bugs.py', '-v'] ... [2022-05-18 03:43:21.645994]
2022-05-18T03:43:22.4658777Z ============================= test session starts ==============================
2022-05-18T03:43:22.4659473Z platform linux -- Python 3.7.13, pytest-7.1.2, pluggy-1.0.0 -- /opt/conda/bin/python
2022-05-18T03:43:22.4674926Z cachedir: .pytest_cache
2022-05-18T03:43:22.4675743Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2022-05-18T03:43:22.4676297Z torch: 1.12.0a0+git3b23752
2022-05-18T03:43:22.4676722Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2022-05-18T03:43:22.4677206Z plugins: hypothesis-4.53.2
2022-05-18T03:43:22.4795303Z collecting ... 
2022-05-18T03:43:22.4795818Z collected 4 items                                                              
2022-05-18T03:43:22.4799430Z 
2022-05-18T03:43:22.5211788Z distributed/pipeline/sync/test_bugs.py::test_python_autograd_function PASSED [ 25%]
2022-05-18T03:43:22.5385397Z distributed/pipeline/sync/test_bugs.py::test_exception_no_hang PASSED    [ 50%]
2022-05-18T03:43:22.5395304Z distributed/pipeline/sync/test_bugs.py::test_tuple_wait SKIPPED (2 c...) [ 75%]
2022-05-18T03:43:22.6406801Z distributed/pipeline/sync/test_bugs.py::test_parallel_randoms PASSED     [100%]
2022-05-18T03:43:22.6407621Z 
2022-05-18T03:43:22.6407764Z =========================== short test summary info ============================
2022-05-18T03:43:22.6408268Z SKIPPED [1] distributed/pipeline/sync/test_bugs.py:73: 2 cuda devices required
2022-05-18T03:43:22.6409286Z ========================= 3 passed, 1 skipped in 0.18s =========================
2022-05-18T03:43:22.7709421Z Running distributed/pipeline/sync/test_checkpoint ... [2022-05-18 03:43:22.770622]
2022-05-18T03:43:22.7710241Z Executing ['/opt/conda/bin/python', '-m', 'pytest', 'distributed/pipeline/sync/test_checkpoint.py', '-v'] ... [2022-05-18 03:43:22.770705]
2022-05-18T03:43:23.5686031Z ============================= test session starts ==============================
2022-05-18T03:43:23.5686471Z platform linux -- Python 3.7.13, pytest-7.1.2, pluggy-1.0.0 -- /opt/conda/bin/python
2022-05-18T03:43:23.5700303Z cachedir: .pytest_cache
2022-05-18T03:43:23.5701127Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2022-05-18T03:43:23.5701486Z torch: 1.12.0a0+git3b23752
2022-05-18T03:43:23.5701732Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2022-05-18T03:43:23.5702022Z plugins: hypothesis-4.53.2
2022-05-18T03:43:23.5833117Z collecting ... 
2022-05-18T03:43:23.5833477Z collected 7 items                                                              
2022-05-18T03:43:23.5836118Z 
2022-05-18T03:43:23.5877550Z distributed/pipeline/sync/test_checkpoint.py::test_serial_checkpoints[cpu] PASSED [ 14%]
2022-05-18T03:43:23.5892145Z distributed/pipeline/sync/test_checkpoint.py::test_not_requires_grad PASSED [ 28%]
2022-05-18T03:43:23.5906550Z distributed/pipeline/sync/test_checkpoint.py::test_not_requires_grad_with_parameter PASSED [ 42%]
2022-05-18T03:43:23.5934997Z distributed/pipeline/sync/test_checkpoint.py::test_random_in_checkpoint[cpu] PASSED [ 57%]
2022-05-18T03:43:23.5949785Z distributed/pipeline/sync/test_checkpoint.py::test_detect_checkpointing_recomputing PASSED [ 71%]
2022-05-18T03:43:23.5961712Z distributed/pipeline/sync/test_checkpoint.py::test_detect_checkpointing_recomputing_without_checkpoint PASSED [ 85%]
2022-05-18T03:43:23.5979713Z distributed/pipeline/sync/test_checkpoint.py::test_non_grad_output PASSED [100%]
2022-05-18T03:43:23.5980904Z 
2022-05-18T03:43:23.5981208Z ============================== 7 passed in 0.03s ===============================
2022-05-18T03:43:23.7254803Z Running distributed/pipeline/sync/test_copy ... [2022-05-18 03:43:23.725083]
2022-05-18T03:43:23.7255291Z Executing ['/opt/conda/bin/python', '-m', 'pytest', 'distributed/pipeline/sync/test_copy.py', '-v'] ... [2022-05-18 03:43:23.725164]
2022-05-18T03:43:24.5172893Z ============================= test session starts ==============================
2022-05-18T03:43:24.5173343Z platform linux -- Python 3.7.13, pytest-7.1.2, pluggy-1.0.0 -- /opt/conda/bin/python
2022-05-18T03:43:24.5188024Z cachedir: .pytest_cache
2022-05-18T03:43:24.5188837Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2022-05-18T03:43:24.5189252Z torch: 1.12.0a0+git3b23752
2022-05-18T03:43:24.5189512Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2022-05-18T03:43:24.5189798Z plugins: hypothesis-4.53.2
2022-05-18T03:43:24.5300927Z collecting ... 
2022-05-18T03:43:24.5301281Z collected 5 items                                                              
2022-05-18T03:43:24.5304609Z 
2022-05-18T03:43:24.5341480Z distributed/pipeline/sync/test_copy.py::test_copy_wait_cpu_cpu PASSED    [ 20%]
2022-05-18T03:43:24.5349934Z distributed/pipeline/sync/test_copy.py::test_copy_wait_cpu_cuda SKIPPED  [ 40%]
2022-05-18T03:43:24.5359007Z distributed/pipeline/sync/test_copy.py::test_copy_wait_cuda_cpu SKIPPED  [ 60%]
2022-05-18T03:43:24.5366147Z distributed/pipeline/sync/test_copy.py::test_copy_wait_cuda_cuda SKIPPED [ 80%]
2022-05-18T03:43:24.5381756Z distributed/pipeline/sync/test_copy.py::test_wait_multiple_tensors PASSED [100%]
2022-05-18T03:43:24.5384280Z 
2022-05-18T03:43:24.5384469Z =========================== short test summary info ============================
2022-05-18T03:43:24.5385079Z SKIPPED [1] distributed/pipeline/sync/test_copy.py:42: cuda required
2022-05-18T03:43:24.5385449Z SKIPPED [1] distributed/pipeline/sync/test_copy.py:49: cuda required
2022-05-18T03:43:24.5385881Z SKIPPED [1] distributed/pipeline/sync/test_copy.py:56: cuda required
2022-05-18T03:43:24.5386289Z ========================= 2 passed, 3 skipped in 0.02s =========================
2022-05-18T03:43:24.6586884Z Running distributed/pipeline/sync/test_deferred_batch_norm ... [2022-05-18 03:43:24.658305]
2022-05-18T03:43:24.6587424Z Executing ['/opt/conda/bin/python', '-m', 'pytest', 'distributed/pipeline/sync/test_deferred_batch_norm.py', '-v'] ... [2022-05-18 03:43:24.658386]
2022-05-18T03:43:25.4566428Z ============================= test session starts ==============================
2022-05-18T03:43:25.4566875Z platform linux -- Python 3.7.13, pytest-7.1.2, pluggy-1.0.0 -- /opt/conda/bin/python
2022-05-18T03:43:25.4581153Z cachedir: .pytest_cache
2022-05-18T03:43:25.4581786Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2022-05-18T03:43:25.4582118Z torch: 1.12.0a0+git3b23752
2022-05-18T03:43:25.4582443Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2022-05-18T03:43:25.4582717Z plugins: hypothesis-4.53.2
2022-05-18T03:43:25.4793522Z collecting ... 
2022-05-18T03:43:25.4793920Z collected 11 items                                                             
2022-05-18T03:43:25.4797007Z 
2022-05-18T03:43:25.5526985Z distributed/pipeline/sync/test_deferred_batch_norm.py::test_transparency[True-1] PASSED [  9%]
2022-05-18T03:43:25.6227850Z distributed/pipeline/sync/test_deferred_batch_norm.py::test_transparency[True-4] PASSED [ 18%]
2022-05-18T03:43:25.6734510Z distributed/pipeline/sync/test_deferred_batch_norm.py::test_transparency[False-1] PASSED [ 27%]
2022-05-18T03:43:25.7192595Z distributed/pipeline/sync/test_deferred_batch_norm.py::test_transparency[False-4] PASSED [ 36%]
2022-05-18T03:43:25.7477770Z distributed/pipeline/sync/test_deferred_batch_norm.py::test_running_stats[0.1] PASSED [ 45%]
2022-05-18T03:43:25.7719603Z distributed/pipeline/sync/test_deferred_batch_norm.py::test_running_stats[None] PASSED [ 54%]
2022-05-18T03:43:25.7737839Z distributed/pipeline/sync/test_deferred_batch_norm.py::test_convert_deferred_batch_norm PASSED [ 63%]
2022-05-18T03:43:25.8161234Z distributed/pipeline/sync/test_deferred_batch_norm.py::test_eval PASSED  [ 72%]
2022-05-18T03:43:25.9848825Z distributed/pipeline/sync/test_deferred_batch_norm.py::test_optimize PASSED [ 81%]
2022-05-18T03:43:26.2156048Z distributed/pipeline/sync/test_deferred_batch_norm.py::test_conv_bn PASSED [ 90%]
2022-05-18T03:43:26.2348798Z distributed/pipeline/sync/test_deferred_batch_norm.py::test_input_requiring_grad PASSED [100%]
2022-05-18T03:43:26.2349447Z 
2022-05-18T03:43:26.2349758Z ============================== 11 passed in 0.78s ==============================
2022-05-18T03:43:26.3617513Z Running distributed/pipeline/sync/test_dependency ... [2022-05-18 03:43:26.361354]
2022-05-18T03:43:26.3618006Z Executing ['/opt/conda/bin/python', '-m', 'pytest', 'distributed/pipeline/sync/test_dependency.py', '-v'] ... [2022-05-18 03:43:26.361435]
2022-05-18T03:43:27.1751448Z ============================= test session starts ==============================
2022-05-18T03:43:27.1751894Z platform linux -- Python 3.7.13, pytest-7.1.2, pluggy-1.0.0 -- /opt/conda/bin/python
2022-05-18T03:43:27.1765567Z cachedir: .pytest_cache
2022-05-18T03:43:27.1766566Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2022-05-18T03:43:27.1767017Z torch: 1.12.0a0+git3b23752
2022-05-18T03:43:27.1767261Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2022-05-18T03:43:27.1767760Z plugins: hypothesis-4.53.2
2022-05-18T03:43:27.1939119Z collecting ... 
2022-05-18T03:43:27.1939575Z collected 6 items                                                              
2022-05-18T03:43:27.1942074Z 
2022-05-18T03:43:27.1951636Z distributed/pipeline/sync/test_dependency.py::test_fork_join SKIPPED     [ 16%]
2022-05-18T03:43:27.1979809Z distributed/pipeline/sync/test_dependency.py::test_fork_join_enable_grad PASSED [ 33%]
2022-05-18T03:43:27.1993271Z distributed/pipeline/sync/test_dependency.py::test_fork_join_no_grad PASSED [ 50%]
2022-05-18T03:43:27.2008346Z distributed/pipeline/sync/test_dependency.py::test_fork_leak PASSED      [ 66%]
2022-05-18T03:43:27.2020624Z distributed/pipeline/sync/test_dependency.py::test_join_when_fork_not_requires_grad PASSED [ 83%]
2022-05-18T03:43:27.2035962Z distributed/pipeline/sync/test_dependency.py::test_join_when_fork_requires_grad PASSED [100%]
2022-05-18T03:43:27.2037577Z 
2022-05-18T03:43:27.2037781Z =========================== short test summary info ============================
2022-05-18T03:43:27.2038259Z SKIPPED [1] distributed/pipeline/sync/test_dependency.py:17: cuda required
2022-05-18T03:43:27.2038881Z ========================= 5 passed, 1 skipped in 0.03s =========================
2022-05-18T03:43:27.3381032Z Running distributed/pipeline/sync/test_inplace ... [2022-05-18 03:43:27.337746]
2022-05-18T03:43:27.3381557Z Executing ['/opt/conda/bin/python', '-m', 'pytest', 'distributed/pipeline/sync/test_inplace.py', '-v'] ... [2022-05-18 03:43:27.337825]
2022-05-18T03:43:28.1306430Z ============================= test session starts ==============================
2022-05-18T03:43:28.1306869Z platform linux -- Python 3.7.13, pytest-7.1.2, pluggy-1.0.0 -- /opt/conda/bin/python
2022-05-18T03:43:28.1321711Z cachedir: .pytest_cache
2022-05-18T03:43:28.1322615Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2022-05-18T03:43:28.1322960Z torch: 1.12.0a0+git3b23752
2022-05-18T03:43:28.1323213Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2022-05-18T03:43:28.1323491Z plugins: hypothesis-4.53.2
2022-05-18T03:43:28.1411063Z collecting ... 
2022-05-18T03:43:28.1411421Z collected 3 items                                                              
2022-05-18T03:43:28.1414688Z 
2022-05-18T03:43:28.1914750Z distributed/pipeline/sync/test_inplace.py::test_inplace_on_requires_grad PASSED [ 33%]
2022-05-18T03:43:28.2080002Z distributed/pipeline/sync/test_inplace.py::test_inplace_on_not_requires_grad XFAIL [ 66%]
2022-05-18T03:43:28.2243330Z distributed/pipeline/sync/test_inplace.py::test_inplace_incorrect_grad XFAIL [100%]
2022-05-18T03:43:28.2245973Z 
2022-05-18T03:43:28.2246201Z =========================== short test summary info ============================
2022-05-18T03:43:28.2246729Z XFAIL distributed/pipeline/sync/test_inplace.py::test_inplace_on_not_requires_grad
2022-05-18T03:43:28.2247351Z XFAIL distributed/pipeline/sync/test_inplace.py::test_inplace_incorrect_grad
2022-05-18T03:43:28.2248040Z ========================= 1 passed, 2 xfailed in 0.09s =========================
2022-05-18T03:43:28.3555739Z Running distributed/pipeline/sync/test_microbatch ... [2022-05-18 03:43:28.355241]
2022-05-18T03:43:28.3556242Z Executing ['/opt/conda/bin/python', '-m', 'pytest', 'distributed/pipeline/sync/test_microbatch.py', '-v'] ... [2022-05-18 03:43:28.355321]
2022-05-18T03:43:29.1560934Z ============================= test session starts ==============================
2022-05-18T03:43:29.1561651Z platform linux -- Python 3.7.13, pytest-7.1.2, pluggy-1.0.0 -- /opt/conda/bin/python
2022-05-18T03:43:29.1576990Z cachedir: .pytest_cache
2022-05-18T03:43:29.1578172Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2022-05-18T03:43:29.1578749Z torch: 1.12.0a0+git3b23752
2022-05-18T03:43:29.1579285Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2022-05-18T03:43:29.1579761Z plugins: hypothesis-4.53.2
2022-05-18T03:43:29.1803207Z collecting ... 
2022-05-18T03:43:29.1803739Z collected 10 items                                                             
2022-05-18T03:43:29.1806058Z 
2022-05-18T03:43:29.1832286Z distributed/pipeline/sync/test_microbatch.py::test_batch_atomic PASSED   [ 10%]
2022-05-18T03:43:29.1844537Z distributed/pipeline/sync/test_microbatch.py::test_batch_non_atomic PASSED [ 20%]
2022-05-18T03:43:29.1855619Z distributed/pipeline/sync/test_microbatch.py::test_batch_call PASSED     [ 30%]
2022-05-18T03:43:29.1867409Z distributed/pipeline/sync/test_microbatch.py::test_batch_setitem_by_index PASSED [ 40%]
2022-05-18T03:43:29.1878611Z distributed/pipeline/sync/test_microbatch.py::test_batch_setitem_by_slice PASSED [ 50%]
2022-05-18T03:43:29.1892605Z distributed/pipeline/sync/test_microbatch.py::test_check PASSED          [ 60%]
2022-05-18T03:43:29.1907304Z distributed/pipeline/sync/test_microbatch.py::test_gather_tensors PASSED [ 70%]
2022-05-18T03:43:29.1918415Z distributed/pipeline/sync/test_microbatch.py::test_gather_tuples PASSED  [ 80%]
2022-05-18T03:43:29.1929925Z distributed/pipeline/sync/test_microbatch.py::test_scatter_tensor PASSED [ 90%]
2022-05-18T03:43:29.1944742Z distributed/pipeline/sync/test_microbatch.py::test_scatter_multiple_tensors PASSED [100%]
2022-05-18T03:43:29.1946098Z 
2022-05-18T03:43:29.1946422Z ============================== 10 passed in 0.04s ==============================
2022-05-18T03:43:29.3290794Z Running distributed/pipeline/sync/test_phony ... [2022-05-18 03:43:29.328655]
2022-05-18T03:43:29.3291304Z Executing ['/opt/conda/bin/python', '-m', 'pytest', 'distributed/pipeline/sync/test_phony.py', '-v'] ... [2022-05-18 03:43:29.328738]
2022-05-18T03:43:30.1274811Z ============================= test session starts ==============================
2022-05-18T03:43:30.1275545Z platform linux -- Python 3.7.13, pytest-7.1.2, pluggy-1.0.0 -- /opt/conda/bin/python
2022-05-18T03:43:30.1290644Z cachedir: .pytest_cache
2022-05-18T03:43:30.1291460Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2022-05-18T03:43:30.1292045Z torch: 1.12.0a0+git3b23752
2022-05-18T03:43:30.1292467Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2022-05-18T03:43:30.1292957Z plugins: hypothesis-4.53.2
2022-05-18T03:43:30.1404189Z collecting ... 
2022-05-18T03:43:30.1404730Z collected 4 items                                                              
2022-05-18T03:43:30.1407876Z 
2022-05-18T03:43:30.1433518Z distributed/pipeline/sync/test_phony.py::test_phony_size PASSED          [ 25%]
2022-05-18T03:43:30.1445568Z distributed/pipeline/sync/test_phony.py::test_phony_requires_grad PASSED [ 50%]
2022-05-18T03:43:30.1456862Z distributed/pipeline/sync/test_phony.py::test_cached_phony PASSED        [ 75%]
2022-05-18T03:43:30.1475264Z distributed/pipeline/sync/test_phony.py::test_phony_in_autograd_function PASSED [100%]
2022-05-18T03:43:30.1476649Z 
2022-05-18T03:43:30.1476889Z ============================== 4 passed in 0.02s ===============================
2022-05-18T03:43:30.2718491Z Running distributed/pipeline/sync/test_pipe ... [2022-05-18 03:43:30.271440]
2022-05-18T03:43:30.2718967Z Executing ['/opt/conda/bin/python', '-m', 'pytest', 'distributed/pipeline/sync/test_pipe.py', '-v'] ... [2022-05-18 03:43:30.271525]
2022-05-18T03:43:31.0693956Z ============================= test session starts ==============================
2022-05-18T03:43:31.0694613Z platform linux -- Python 3.7.13, pytest-7.1.2, pluggy-1.0.0 -- /opt/conda/bin/python
2022-05-18T03:43:31.0708872Z cachedir: .pytest_cache
2022-05-18T03:43:31.0709610Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2022-05-18T03:43:31.0710207Z torch: 1.12.0a0+git3b23752
2022-05-18T03:43:31.0710491Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2022-05-18T03:43:31.0710765Z plugins: hypothesis-4.53.2
2022-05-18T03:43:31.1376541Z collecting ... 
2022-05-18T03:43:31.1376984Z collected 56 items                                                             
2022-05-18T03:43:31.1379624Z 
2022-05-18T03:43:31.1410852Z distributed/pipeline/sync/test_pipe.py::test_pipe_without_rpc PASSED     [  1%]
2022-05-18T03:43:31.1996826Z distributed/pipeline/sync/test_pipe.py::test_parameters PASSED           [  3%]
2022-05-18T03:43:31.2222462Z distributed/pipeline/sync/test_pipe.py::test_public_attrs PASSED         [  5%]
2022-05-18T03:43:31.2400484Z distributed/pipeline/sync/test_pipe.py::test_sequential_like PASSED      [  7%]
2022-05-18T03:43:31.2539269Z distributed/pipeline/sync/test_pipe.py::test_chunks_less_than_1 PASSED   [  8%]
2022-05-18T03:43:31.2721449Z distributed/pipeline/sync/test_pipe.py::test_batch_size_indivisible PASSED [ 10%]
2022-05-18T03:43:31.2882906Z distributed/pipeline/sync/test_pipe.py::test_batch_size_small PASSED     [ 12%]
2022-05-18T03:43:31.3052969Z distributed/pipeline/sync/test_pipe.py::test_checkpoint_mode PASSED      [ 14%]
2022-05-18T03:43:31.3192293Z distributed/pipeline/sync/test_pipe.py::test_checkpoint_mode_invalid PASSED [ 16%]
2022-05-18T03:43:31.3383280Z distributed/pipeline/sync/test_pipe.py::test_checkpoint_mode_when_chunks_1 PASSED [ 17%]
2022-05-18T03:43:31.3600210Z distributed/pipeline/sync/test_pipe.py::test_checkpoint_eval PASSED      [ 19%]
2022-05-18T03:43:31.3800472Z distributed/pipeline/sync/test_pipe.py::test_checkpoint_non_float_input PASSED [ 21%]
2022-05-18T03:43:31.3946353Z distributed/pipeline/sync/test_pipe.py::test_no_grad PASSED              [ 23%]
2022-05-18T03:43:31.4087651Z distributed/pipeline/sync/test_pipe.py::test_exception PASSED            [ 25%]
2022-05-18T03:43:31.6274828Z distributed/pipeline/sync/test_pipe.py::test_exception_early_stop_asap PASSED [ 26%]
2022-05-18T03:43:31.6521685Z distributed/pipeline/sync/test_pipe.py::test_nested_input PASSED         [ 28%]
2022-05-18T03:43:31.6707152Z distributed/pipeline/sync/test_pipe.py::test_input_pair PASSED           [ 30%]
2022-05-18T03:43:31.6868288Z distributed/pipeline/sync/test_pipe.py::test_multi_sequence_input PASSED [ 32%]
2022-05-18T03:43:31.7079815Z distributed/pipeline/sync/test_pipe.py::test_input_singleton PASSED      [ 33%]
2022-05-18T03:43:31.7280004Z distributed/pipeline/sync/test_pipe.py::test_input_varargs PASSED        [ 35%]
2022-05-18T03:43:31.7479079Z distributed/pipeline/sync/test_pipe.py::test_non_tensor PASSED           [ 37%]
2022-05-18T03:43:31.7639179Z distributed/pipeline/sync/test_pipe.py::test_non_tensor_sequence PASSED  [ 39%]
2022-05-18T03:43:31.7840771Z distributed/pipeline/sync/test_pipe.py::test_valid_non_tensor[never] PASSED [ 41%]
2022-05-18T03:43:31.8080664Z distributed/pipeline/sync/test_pipe.py::test_valid_non_tensor[always] PASSED [ 42%]
2022-05-18T03:43:31.8319784Z distributed/pipeline/sync/test_pipe.py::test_valid_non_tensor[except_last] PASSED [ 44%]
2022-05-18T03:43:31.8519461Z distributed/pipeline/sync/test_pipe.py::test_no_tensor_output[never] PASSED [ 46%]
2022-05-18T03:43:31.8681187Z distributed/pipeline/sync/test_pipe.py::test_no_tensor_output[always] PASSED [ 48%]
2022-05-18T03:43:31.8841181Z distributed/pipeline/sync/test_pipe.py::test_no_tensor_output[except_last] PASSED [ 50%]
2022-05-18T03:43:31.9071905Z distributed/pipeline/sync/test_pipe.py::test_uneven_batch_size[never] PASSED [ 51%]
2022-05-18T03:43:31.9319482Z distributed/pipeline/sync/test_pipe.py::test_uneven_batch_size[always] PASSED [ 53%]
2022-05-18T03:43:31.9519347Z distributed/pipeline/sync/test_pipe.py::test_uneven_batch_size[except_last] PASSED [ 55%]
2022-05-18T03:43:31.9749125Z distributed/pipeline/sync/test_pipe.py::test_no_chunk[never] PASSED      [ 57%]
2022-05-18T03:43:31.9959802Z distributed/pipeline/sync/test_pipe.py::test_no_chunk[always] PASSED     [ 58%]
2022-05-18T03:43:32.0114825Z distributed/pipeline/sync/test_pipe.py::test_no_chunk[except_last] PASSED [ 60%]
2022-05-18T03:43:32.0368860Z distributed/pipeline/sync/test_pipe.py::test_deferred_batch_norm[never] PASSED [ 62%]
2022-05-18T03:43:32.0561598Z distributed/pipeline/sync/test_pipe.py::test_deferred_batch_norm[always] PASSED [ 64%]
2022-05-18T03:43:32.0760997Z distributed/pipeline/sync/test_pipe.py::test_deferred_batch_norm[except_last] PASSED [ 66%]
2022-05-18T03:43:32.0999718Z distributed/pipeline/sync/test_pipe.py::test_deferred_batch_norm_params[never] PASSED [ 67%]
2022-05-18T03:43:32.1200589Z distributed/pipeline/sync/test_pipe.py::test_deferred_batch_norm_params[always] PASSED [ 69%]
2022-05-18T03:43:32.1399041Z distributed/pipeline/sync/test_pipe.py::test_devices PASSED              [ 71%]
2022-05-18T03:43:32.1599029Z distributed/pipeline/sync/test_pipe.py::test_partitions PASSED           [ 73%]
2022-05-18T03:43:32.1607497Z distributed/pipeline/sync/test_pipe.py::test_merged_partitions SKIPPED   [ 75%]
2022-05-18T03:43:32.1755876Z distributed/pipeline/sync/test_pipe.py::test_deny_moving PASSED          [ 76%]
2022-05-18T03:43:32.1959060Z distributed/pipeline/sync/test_pipe.py::test_empty_module PASSED         [ 78%]
2022-05-18T03:43:32.2150928Z distributed/pipeline/sync/test_pipe.py::test_named_children PASSED       [ 80%]
2022-05-18T03:43:32.2286509Z distributed/pipeline/sync/test_pipe.py::test_verify_module_non_sequential PASSED [ 82%]
2022-05-18T03:43:32.2457631Z distributed/pipeline/sync/test_pipe.py::test_verify_module_duplicate_children PASSED [ 83%]
2022-05-18T03:43:32.2466123Z distributed/pipeline/sync/test_pipe.py::test_verify_module_params_on_same_device SKIPPED [ 85%]
2022-05-18T03:43:32.2473973Z distributed/pipeline/sync/test_pipe.py::test_verify_nested_modules SKIPPED [ 87%]
2022-05-18T03:43:32.2640051Z distributed/pipeline/sync/test_pipe.py::test_verify_module_duplicate_parameters_on_same_device PASSED [ 89%]
2022-05-18T03:43:32.5845455Z distributed/pipeline/sync/test_pipe.py::test_forward_lockstep PASSED     [ 91%]
2022-05-18T03:43:32.5853727Z distributed/pipeline/sync/test_pipe.py::test_multiple_inputs[never] SKIPPED [ 92%]
2022-05-18T03:43:32.5861503Z distributed/pipeline/sync/test_pipe.py::test_multiple_inputs[always] SKIPPED [ 94%]
2022-05-18T03:43:32.5869498Z distributed/pipeline/sync/test_pipe.py::test_multiple_inputs[except_last] SKIPPED [ 96%]
2022-05-18T03:43:32.5877007Z distributed/pipeline/sync/test_pipe.py::test_inputs_wrong_device SKIPPED [ 98%]
2022-05-18T03:43:32.5888922Z distributed/pipeline/sync/test_pipe.py::test_with_device_wrapper SKIPPED [100%]
2022-05-18T03:43:32.5889950Z 
2022-05-18T03:43:32.5890227Z =============================== warnings summary ===============================
2022-05-18T03:43:32.5890704Z test/distributed/pipeline/sync/test_pipe.py::test_batch_size_indivisible
2022-05-18T03:43:32.5891372Z test/distributed/pipeline/sync/test_pipe.py::test_batch_size_small
2022-05-18T03:43:32.5892249Z   /opt/conda/lib/python3.7/site-packages/_pytest/python.py:192: PytestRemovedIn8Warning: Passing None has been deprecated.
2022-05-18T03:43:32.5893168Z   See https://docs.pytest.org/en/latest/how-to/capture-warnings.html#additional-use-cases-of-warnings-in-tests for alternatives in common use cases.
2022-05-18T03:43:32.5893771Z     result = testfunction(**testargs)
2022-05-18T03:43:32.5894010Z 
2022-05-18T03:43:32.5894426Z -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
2022-05-18T03:43:32.5896578Z =========================== short test summary info ============================
2022-05-18T03:43:32.5897263Z SKIPPED [1] distributed/pipeline/sync/test_pipe.py:590: cuda required
2022-05-18T03:43:32.5897703Z SKIPPED [1] distributed/pipeline/sync/test_pipe.py:688: cuda required
2022-05-18T03:43:32.5898020Z SKIPPED [1] distributed/pipeline/sync/test_pipe.py:706: Need atleast two GPUs
2022-05-18T03:43:32.5898361Z SKIPPED [3] distributed/pipeline/sync/test_pipe.py:766: cuda required
2022-05-18T03:43:32.5898660Z SKIPPED [1] distributed/pipeline/sync/test_pipe.py:782: Need atleast two GPUs
2022-05-18T03:43:32.5898975Z SKIPPED [1] distributed/pipeline/sync/test_pipe.py:800: Need atleast two GPUs
2022-05-18T03:43:32.5899420Z ================== 48 passed, 8 skipped, 2 warnings in 1.52s ===================
2022-05-18T03:43:32.7240455Z Running distributed/pipeline/sync/test_pipeline ... [2022-05-18 03:43:32.723697]
2022-05-18T03:43:32.7240953Z Executing ['/opt/conda/bin/python', '-m', 'pytest', 'distributed/pipeline/sync/test_pipeline.py', '-v'] ... [2022-05-18 03:43:32.723782]
2022-05-18T03:43:33.5378969Z ============================= test session starts ==============================
2022-05-18T03:43:33.5379438Z platform linux -- Python 3.7.13, pytest-7.1.2, pluggy-1.0.0 -- /opt/conda/bin/python
2022-05-18T03:43:33.5393674Z cachedir: .pytest_cache
2022-05-18T03:43:33.5394320Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2022-05-18T03:43:33.5394717Z torch: 1.12.0a0+git3b23752
2022-05-18T03:43:33.5395013Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2022-05-18T03:43:33.5395284Z plugins: hypothesis-4.53.2
2022-05-18T03:43:33.5503823Z collecting ... 
2022-05-18T03:43:33.5504230Z collected 1 item                                                               
2022-05-18T03:43:33.5507248Z 
2022-05-18T03:43:33.5534241Z distributed/pipeline/sync/test_pipeline.py::test_clock_cycles PASSED     [100%]
2022-05-18T03:43:33.5535323Z 
2022-05-18T03:43:33.5535555Z ============================== 1 passed in 0.02s ===============================
2022-05-18T03:43:33.6727978Z Running distributed/pipeline/sync/test_stream ... [2022-05-18 03:43:33.672423]
2022-05-18T03:43:33.6728456Z Executing ['/opt/conda/bin/python', '-m', 'pytest', 'distributed/pipeline/sync/test_stream.py', '-v'] ... [2022-05-18 03:43:33.672503]
2022-05-18T03:43:34.4690165Z ============================= test session starts ==============================
2022-05-18T03:43:34.4690664Z platform linux -- Python 3.7.13, pytest-7.1.2, pluggy-1.0.0 -- /opt/conda/bin/python
2022-05-18T03:43:34.4705032Z cachedir: .pytest_cache
2022-05-18T03:43:34.4705753Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2022-05-18T03:43:34.4706123Z torch: 1.12.0a0+git3b23752
2022-05-18T03:43:34.4706366Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2022-05-18T03:43:34.4706655Z plugins: hypothesis-4.53.2
2022-05-18T03:43:34.4923682Z collecting ... 
2022-05-18T03:43:34.4924025Z collected 19 items                                                             
2022-05-18T03:43:34.4926766Z 
2022-05-18T03:43:34.4949623Z distributed/pipeline/sync/test_stream.py::TestNewStream::test_new_stream_cpu PASSED [  5%]
2022-05-18T03:43:34.4957888Z distributed/pipeline/sync/test_stream.py::TestNewStream::test_new_stream_cuda SKIPPED [ 10%]
2022-05-18T03:43:34.4969210Z distributed/pipeline/sync/test_stream.py::TestCurrentStream::test_current_stream_cpu PASSED [ 15%]
2022-05-18T03:43:34.4976643Z distributed/pipeline/sync/test_stream.py::TestCurrentStream::test_current_stream_cuda SKIPPED [ 21%]
2022-05-18T03:43:34.4989682Z distributed/pipeline/sync/test_stream.py::TestDefaultStream::test_default_stream_cpu PASSED [ 26%]
2022-05-18T03:43:34.4997191Z distributed/pipeline/sync/test_stream.py::TestDefaultStream::test_default_stream_cuda SKIPPED [ 31%]
2022-05-18T03:43:34.5008244Z distributed/pipeline/sync/test_stream.py::TestUseDevice::test_use_device_cpu PASSED [ 36%]
2022-05-18T03:43:34.5015732Z distributed/pipeline/sync/test_stream.py::TestUseDevice::test_use_device_cuda SKIPPED [ 42%]
2022-05-18T03:43:34.5027862Z distributed/pipeline/sync/test_stream.py::TestUseStream::test_use_stream_cpu PASSED [ 47%]
2022-05-18T03:43:34.5035986Z distributed/pipeline/sync/test_stream.py::TestUseStream::test_use_stream_cuda SKIPPED [ 52%]
2022-05-18T03:43:34.5047077Z distributed/pipeline/sync/test_stream.py::TestGetDevice::test_get_device_cpu PASSED [ 57%]
2022-05-18T03:43:34.5054553Z distributed/pipeline/sync/test_stream.py::TestGetDevice::test_get_device_cuda SKIPPED [ 63%]
2022-05-18T03:43:34.5072409Z distributed/pipeline/sync/test_stream.py::TestWaitStream::test_wait_stream_cpu_cpu PASSED [ 68%]
2022-05-18T03:43:34.5079761Z distributed/pipeline/sync/test_stream.py::TestWaitStream::test_wait_stream_cpu_cuda SKIPPED [ 73%]
2022-05-18T03:43:34.5087257Z distributed/pipeline/sync/test_stream.py::TestWaitStream::test_wait_stream_cuda_cpu SKIPPED [ 78%]
2022-05-18T03:43:34.5094850Z distributed/pipeline/sync/test_stream.py::TestWaitStream::test_wait_stream_cuda_cuda SKIPPED [ 84%]
2022-05-18T03:43:34.5107063Z distributed/pipeline/sync/test_stream.py::TestRecordStream::test_record_stream_cpu PASSED [ 89%]
2022-05-18T03:43:34.5114539Z distributed/pipeline/sync/test_stream.py::TestRecordStream::test_record_stream_cuda SKIPPED [ 94%]
2022-05-18T03:43:34.5127545Z distributed/pipeline/sync/test_stream.py::TestRecordStream::test_record_stream_shifted_view SKIPPED [100%]
2022-05-18T03:43:34.5132838Z 
2022-05-18T03:43:34.5133214Z =========================== short test summary info ============================
2022-05-18T03:43:34.5133567Z SKIPPED [1] distributed/pipeline/sync/test_stream.py:33: cuda required
2022-05-18T03:43:34.5133993Z SKIPPED [1] distributed/pipeline/sync/test_stream.py:45: cuda required
2022-05-18T03:43:34.5134328Z SKIPPED [1] distributed/pipeline/sync/test_stream.py:57: cuda required
2022-05-18T03:43:34.5134757Z SKIPPED [1] distributed/pipeline/sync/test_stream.py:69: cuda required
2022-05-18T03:43:34.5135283Z SKIPPED [1] distributed/pipeline/sync/test_stream.py:80: cuda required
2022-05-18T03:43:34.5135780Z SKIPPED [1] distributed/pipeline/sync/test_stream.py:91: cuda required
2022-05-18T03:43:34.5136071Z SKIPPED [1] distributed/pipeline/sync/test_stream.py:114: cuda required
2022-05-18T03:43:34.5136366Z SKIPPED [1] distributed/pipeline/sync/test_stream.py:120: cuda required
2022-05-18T03:43:34.5136661Z SKIPPED [1] distributed/pipeline/sync/test_stream.py:126: cuda required
2022-05-18T03:43:34.5136954Z SKIPPED [1] distributed/pipeline/sync/test_stream.py:139: cuda required
2022-05-18T03:43:34.5137229Z SKIPPED [1] distributed/pipeline/sync/test_stream.py:169: cuda required
2022-05-18T03:43:34.5137801Z ======================== 8 passed, 11 skipped in 0.04s =========================
2022-05-18T03:43:34.6348769Z Running distributed/pipeline/sync/test_transparency ... [2022-05-18 03:43:34.634500]
2022-05-18T03:43:34.6349298Z Executing ['/opt/conda/bin/python', '-m', 'pytest', 'distributed/pipeline/sync/test_transparency.py', '-v'] ... [2022-05-18 03:43:34.634585]
2022-05-18T03:43:35.4299287Z ============================= test session starts ==============================
2022-05-18T03:43:35.4299754Z platform linux -- Python 3.7.13, pytest-7.1.2, pluggy-1.0.0 -- /opt/conda/bin/python
2022-05-18T03:43:35.4313809Z cachedir: .pytest_cache
2022-05-18T03:43:35.4314519Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2022-05-18T03:43:35.4314967Z torch: 1.12.0a0+git3b23752
2022-05-18T03:43:35.4315235Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2022-05-18T03:43:35.4315577Z plugins: hypothesis-4.53.2
2022-05-18T03:43:35.4398230Z collecting ... 
2022-05-18T03:43:35.4398622Z collected 1 item                                                               
2022-05-18T03:43:35.4401748Z 
2022-05-18T03:43:35.4918906Z distributed/pipeline/sync/test_transparency.py::test_simple_linears PASSED [100%]
2022-05-18T03:43:35.4919371Z 
2022-05-18T03:43:35.4919679Z ============================== 1 passed in 0.06s ===============================
2022-05-18T03:43:35.6212760Z Running distributed/pipeline/sync/test_worker ... [2022-05-18 03:43:35.620943]
2022-05-18T03:43:35.6213247Z Executing ['/opt/conda/bin/python', '-m', 'pytest', 'distributed/pipeline/sync/test_worker.py', '-v'] ... [2022-05-18 03:43:35.621023]
2022-05-18T03:43:36.4143512Z ============================= test session starts ==============================
2022-05-18T03:43:36.4143964Z platform linux -- Python 3.7.13, pytest-7.1.2, pluggy-1.0.0 -- /opt/conda/bin/python
2022-05-18T03:43:36.4158927Z cachedir: .pytest_cache
2022-05-18T03:43:36.4159601Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2022-05-18T03:43:36.4159995Z torch: 1.12.0a0+git3b23752
2022-05-18T03:43:36.4160243Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2022-05-18T03:43:36.4160533Z plugins: hypothesis-4.53.2
2022-05-18T03:43:36.4305687Z collecting ... 
2022-05-18T03:43:36.4306035Z collected 6 items                                                              
2022-05-18T03:43:36.4308977Z 
2022-05-18T03:43:36.4343842Z distributed/pipeline/sync/test_worker.py::test_compute_multithreading PASSED [ 16%]
2022-05-18T03:43:36.4359984Z distributed/pipeline/sync/test_worker.py::test_compute_success PASSED    [ 33%]
2022-05-18T03:43:36.4373544Z distributed/pipeline/sync/test_worker.py::test_compute_exception PASSED  [ 50%]
2022-05-18T03:43:36.4391409Z distributed/pipeline/sync/test_worker.py::test_grad_mode[True] PASSED    [ 66%]
2022-05-18T03:43:36.4405540Z distributed/pipeline/sync/test_worker.py::test_grad_mode[False] PASSED   [ 83%]
2022-05-18T03:43:36.4425348Z distributed/pipeline/sync/test_worker.py::test_worker_per_device PASSED  [100%]
2022-05-18T03:43:36.4426697Z 
2022-05-18T03:43:36.4426965Z ============================== 6 passed in 0.03s ===============================
2022-05-18T03:43:36.5666473Z Running distributed/rpc/cuda/test_tensorpipe_agent ... [2022-05-18 03:43:36.566252]
2022-05-18T03:43:36.5667072Z Executing ['/opt/conda/bin/python', 'distributed/rpc/cuda/test_tensorpipe_agent.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-05-18 03:43:36.566350]
2022-05-18T03:43:37.1156600Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprdmb13tk
2022-05-18T03:43:37.1157410Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprdmb13tk/_remote_module_non_scriptable.py
2022-05-18T03:43:37.3646775Z <unittest.suite.TestSuite tests=[<__main__.TensorPipeCudaDdpComparisonTest testMethod=test_ddp_dist_autograd_local_vs_remote_gpu>]>
2022-05-18T03:43:37.3647649Z test_ddp_dist_autograd_local_vs_remote_gpu (__main__.TensorPipeCudaDdpComparisonTest)
2022-05-18T03:43:37.3648681Z <unittest.suite.TestSuite tests=[<__main__.TensorPipeCudaDistAutogradTest testMethod=test_gpu_simple>, <__main__.TensorPipeCudaDistAutogradTest testMethod=test_gpu_to_cpu_continuation>, <__main__.TensorPipeCudaDistAutogradTest testMethod=test_gpu_to_cpu_continuation_gpu_root>]>
2022-05-18T03:43:37.3649619Z test_gpu_simple (__main__.TensorPipeCudaDistAutogradTest)
2022-05-18T03:43:37.3650116Z test_gpu_to_cpu_continuation (__main__.TensorPipeCudaDistAutogradTest)
2022-05-18T03:43:37.3650556Z test_gpu_to_cpu_continuation_gpu_root (__main__.TensorPipeCudaDistAutogradTest)
2022-05-18T03:43:37.3651736Z <unittest.suite.TestSuite tests=[<__main__.TensorPipeCudaRemoteModuleTest testMethod=test_input_moved_to_cuda_device>, <__main__.TensorPipeCudaRemoteModuleTest testMethod=test_input_moved_to_cuda_device_script>, <__main__.TensorPipeCudaRemoteModuleTest testMethod=test_invalid_devices>, <__main__.TensorPipeCudaRemoteModuleTest testMethod=test_valid_device>]>
2022-05-18T03:43:37.3652852Z test_input_moved_to_cuda_device (__main__.TensorPipeCudaRemoteModuleTest)
2022-05-18T03:43:37.3653422Z test_input_moved_to_cuda_device_script (__main__.TensorPipeCudaRemoteModuleTest)
2022-05-18T03:43:37.3654018Z test_invalid_devices (__main__.TensorPipeCudaRemoteModuleTest)
2022-05-18T03:43:37.3654554Z test_valid_device (__main__.TensorPipeCudaRemoteModuleTest)
2022-05-18T03:43:37.3655203Z <unittest.suite.TestSuite tests=[<__main__.TensorPipeCudaRpcTest testMethod=test_profiler_remote_cuda>]>
2022-05-18T03:43:37.3655825Z test_profiler_remote_cuda (__main__.TensorPipeCudaRpcTest)
2022-05-18T03:43:37.3657251Z <unittest.suite.TestSuite tests=[<__main__.TensorPipePipeWithDDPTest testMethod=test_basic_gloo_ckpt_always>, <__main__.TensorPipePipeWithDDPTest testMethod=test_basic_gloo_ckpt_except_last>, <__main__.TensorPipePipeWithDDPTest testMethod=test_basic_gloo_ckpt_never>, <__main__.TensorPipePipeWithDDPTest testMethod=test_basic_gloo_ckpt_never_find_unused>, <__main__.TensorPipePipeWithDDPTest testMethod=test_basic_nccl_ckpt_always>, <__main__.TensorPipePipeWithDDPTest testMethod=test_basic_nccl_ckpt_except_last>, <__main__.TensorPipePipeWithDDPTest testMethod=test_basic_nccl_ckpt_never>, <__main__.TensorPipePipeWithDDPTest testMethod=test_basic_nccl_ckpt_never_find_unused>]>
2022-05-18T03:43:37.3658194Z test_basic_gloo_ckpt_always (__main__.TensorPipePipeWithDDPTest)
2022-05-18T03:43:37.3658498Z test_basic_gloo_ckpt_except_last (__main__.TensorPipePipeWithDDPTest)
2022-05-18T03:43:37.3658810Z test_basic_gloo_ckpt_never (__main__.TensorPipePipeWithDDPTest)
2022-05-18T03:43:37.3659128Z test_basic_gloo_ckpt_never_find_unused (__main__.TensorPipePipeWithDDPTest)
2022-05-18T03:43:37.3659444Z test_basic_nccl_ckpt_always (__main__.TensorPipePipeWithDDPTest)
2022-05-18T03:43:37.3659744Z test_basic_nccl_ckpt_except_last (__main__.TensorPipePipeWithDDPTest)
2022-05-18T03:43:37.3660050Z test_basic_nccl_ckpt_never (__main__.TensorPipePipeWithDDPTest)
2022-05-18T03:43:37.3660359Z test_basic_nccl_ckpt_never_find_unused (__main__.TensorPipePipeWithDDPTest)
2022-05-18T03:43:37.3672578Z <unittest.suite.TestSuite tests=[<__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_async_execution_nested_with_cuda_future>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_async_execution_with_cuda_future>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_callback_changes_devices>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_can_extract_cuda_sparse_tensor>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_can_extract_cuda_tensor>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_can_extract_custom_class_with_cuda_sparse_tensor>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_can_extract_custom_class_with_cuda_tensor>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_can_extract_list_with_cuda_sparse_tensor>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_can_extract_list_with_cuda_tensor>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_device_as_device>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_device_as_int>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_device_as_str>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_device_not_cuda>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_modify_tensor_inplace>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_replace_tensor>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_value_on_bad_device>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_custom_stream>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_custom_stream_multi>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_custom_stream_nested>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_custom_stream_nested_multi>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_cpu>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_cpu_to_gpu_default>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_cpu_to_gpu_non_default>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_default>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_default_to_non_default>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_1>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_2>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_3>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_4>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_5>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_6>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_7>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_8>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_self_1>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_self_2>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_self_3>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_self_4>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_self_5>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_self_6>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_self_7>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_self_8>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_non_default>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_non_default_to_default>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_to_cpu_default>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_to_cpu_non_default>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_gpu>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_in_options>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_invalid_max_local_device>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_invalid_max_remote_device>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_invalid_min_device>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_many_to_one>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_missing_config>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_missing_config_loop>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_missing_config_not_timeout>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_missing_config_remote>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_missing_config_remote_response>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_missing_config_response>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_missing_config_response_loop>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_multi_gpu>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_multi_gpu_self>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_one_to_many>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_remote>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_return_to_gpu>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_return_to_gpu_self>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_wrong_worker_name>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_mismatch>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_devices_option_mismatch>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_devices_option_mismatch_reverse>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_meta_multiple_tensors>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_owner_rref_forward_synchronization1>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_owner_rref_forward_synchronization2>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_owner_rref_forward_synchronization3>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_owner_rref_forward_synchronization4>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_as_arg_synchronization1>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_as_arg_synchronization2>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_as_arg_synchronization3>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_as_arg_synchronization4>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_as_arg_synchronization5>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_forward_synchronization1>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_forward_synchronization2>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_forward_synchronization3>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_forward_synchronization4>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_to_here_synchronization1>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_to_here_synchronization2>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_to_here_synchronization3>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_to_here_synchronization4>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_with_unpickleable_attributes>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_tensor_view_as_return_value>]>
2022-05-18T03:43:37.3683864Z test_async_execution_nested_with_cuda_future (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-05-18T03:43:37.3684252Z test_async_execution_with_cuda_future (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-05-18T03:43:37.3684641Z test_cuda_future_callback_changes_devices (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-05-18T03:43:37.3685069Z test_cuda_future_can_extract_cuda_sparse_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-05-18T03:43:37.3685491Z test_cuda_future_can_extract_cuda_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-05-18T03:43:37.3685889Z test_cuda_future_can_extract_custom_class_with_cuda_sparse_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-05-18T03:43:37.3686321Z test_cuda_future_can_extract_custom_class_with_cuda_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-05-18T03:43:37.3686742Z test_cuda_future_can_extract_list_with_cuda_sparse_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-05-18T03:43:37.3687138Z test_cuda_future_can_extract_list_with_cuda_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-05-18T03:43:37.3687520Z test_cuda_future_device_as_device (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-05-18T03:43:37.3687884Z test_cuda_future_device_as_int (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-05-18T03:43:37.3688244Z test_cuda_future_device_as_str (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-05-18T03:43:37.3688595Z test_cuda_future_device_not_cuda (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-05-18T03:43:37.3688969Z test_cuda_future_modify_tensor_inplace (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-05-18T03:43:37.3689341Z test_cuda_future_replace_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-05-18T03:43:37.3689696Z test_cuda_future_value_on_bad_device (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-05-18T03:43:37.3690048Z test_custom_stream (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-05-18T03:43:37.3690387Z test_custom_stream_multi (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-05-18T03:43:37.3690735Z test_custom_stream_nested (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-05-18T03:43:37.3691077Z test_custom_stream_nested_multi (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-05-18T03:43:37.3691426Z test_device_map_cpu (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-05-18T03:43:37.3691786Z test_device_map_cpu_to_gpu_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-05-18T03:43:37.3692150Z test_device_map_cpu_to_gpu_non_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-05-18T03:43:37.3692517Z test_device_map_gpu_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-05-18T03:43:37.3692888Z test_device_map_gpu_default_to_non_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-05-18T03:43:37.3693256Z test_device_map_gpu_mixed_1 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-05-18T03:43:37.3693592Z test_device_map_gpu_mixed_2 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-05-18T03:43:37.3693941Z test_device_map_gpu_mixed_3 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-05-18T03:43:37.3694290Z test_device_map_gpu_mixed_4 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-05-18T03:43:37.3694623Z test_device_map_gpu_mixed_5 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-05-18T03:43:37.3694977Z test_device_map_gpu_mixed_6 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-05-18T03:43:37.3695326Z test_device_map_gpu_mixed_7 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-05-18T03:43:37.3695673Z test_device_map_gpu_mixed_8 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-05-18T03:43:37.3696011Z test_device_map_gpu_mixed_self_1 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-05-18T03:43:37.3696370Z test_device_map_gpu_mixed_self_2 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-05-18T03:43:37.3696729Z test_device_map_gpu_mixed_self_3 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-05-18T03:43:37.3697069Z test_device_map_gpu_mixed_self_4 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-05-18T03:43:37.3697420Z test_device_map_gpu_mixed_self_5 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-05-18T03:43:37.3697776Z test_device_map_gpu_mixed_self_6 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-05-18T03:43:37.3698168Z test_device_map_gpu_mixed_self_7 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-05-18T03:43:37.3698512Z test_device_map_gpu_mixed_self_8 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-05-18T03:43:37.3698899Z test_device_map_gpu_non_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-05-18T03:43:37.3699277Z test_device_map_gpu_non_default_to_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-05-18T03:43:37.3699644Z test_device_map_gpu_to_cpu_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-05-18T03:43:37.3700017Z test_device_map_gpu_to_cpu_non_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-05-18T03:43:37.3700373Z test_device_maps_gpu (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-05-18T03:43:37.3700721Z test_device_maps_in_options (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-05-18T03:43:37.3701074Z test_device_maps_invalid_max_local_device (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-05-18T03:43:37.3701458Z test_device_maps_invalid_max_remote_device (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-05-18T03:43:37.3701839Z test_device_maps_invalid_min_device (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-05-18T03:43:37.3702189Z test_device_maps_many_to_one (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-05-18T03:43:37.3702546Z test_device_maps_missing_config (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-05-18T03:43:37.3703036Z test_device_maps_missing_config_loop (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-05-18T03:43:37.3703432Z test_device_maps_missing_config_not_timeout (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-05-18T03:43:37.3703806Z test_device_maps_missing_config_remote (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-05-18T03:43:37.3704253Z test_device_maps_missing_config_remote_response (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-05-18T03:43:37.3704650Z test_device_maps_missing_config_response (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-05-18T03:43:37.3705034Z test_device_maps_missing_config_response_loop (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-05-18T03:43:37.3705415Z test_device_maps_multi_gpu (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-05-18T03:43:37.3705783Z test_device_maps_multi_gpu_self (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-05-18T03:43:37.3706144Z test_device_maps_one_to_many (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-05-18T03:43:37.3706484Z test_device_maps_remote (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-05-18T03:43:37.3706840Z test_device_maps_return_to_gpu (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-05-18T03:43:37.3707210Z test_device_maps_return_to_gpu_self (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-05-18T03:43:37.3707570Z test_device_maps_wrong_worker_name (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-05-18T03:43:37.3707925Z test_device_mismatch (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-05-18T03:43:37.3708275Z test_devices_option_mismatch (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-05-18T03:43:37.3708645Z test_devices_option_mismatch_reverse (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-05-18T03:43:37.3708997Z test_meta_multiple_tensors (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-05-18T03:43:37.3709370Z test_owner_rref_forward_synchronization1 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-05-18T03:43:37.3709762Z test_owner_rref_forward_synchronization2 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-05-18T03:43:37.3710135Z test_owner_rref_forward_synchronization3 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-05-18T03:43:37.3710517Z test_owner_rref_forward_synchronization4 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-05-18T03:43:37.3710896Z test_rref_as_arg_synchronization1 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-05-18T03:43:37.3711264Z test_rref_as_arg_synchronization2 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-05-18T03:43:37.3711616Z test_rref_as_arg_synchronization3 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-05-18T03:43:37.3712061Z test_rref_as_arg_synchronization4 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-05-18T03:43:37.3712472Z test_rref_as_arg_synchronization5 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-05-18T03:43:37.3712835Z test_rref_forward_synchronization1 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-05-18T03:43:37.3713213Z test_rref_forward_synchronization2 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-05-18T03:43:37.3713593Z test_rref_forward_synchronization3 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-05-18T03:43:37.3713969Z test_rref_forward_synchronization4 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-05-18T03:43:37.3714323Z test_rref_to_here_synchronization1 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-05-18T03:43:37.3714690Z test_rref_to_here_synchronization2 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-05-18T03:43:37.3715060Z test_rref_to_here_synchronization3 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-05-18T03:43:37.3715418Z test_rref_to_here_synchronization4 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-05-18T03:43:37.3715799Z test_rref_with_unpickleable_attributes (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-05-18T03:43:37.3716175Z test_tensor_view_as_return_value (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-05-18T03:43:37.3716869Z <unittest.suite.TestSuite tests=[<__main__.TensorPipeTensorPipeCudaDistAutogradTest testMethod=test_device_maps_backward_pass>, <__main__.TensorPipeTensorPipeCudaDistAutogradTest testMethod=test_dist_autograd_sync_streams>, <__main__.TensorPipeTensorPipeCudaDistAutogradTest testMethod=test_gradients_synchronizations>]>
2022-05-18T03:43:37.3717548Z test_device_maps_backward_pass (__main__.TensorPipeTensorPipeCudaDistAutogradTest)
2022-05-18T03:43:37.3717936Z test_dist_autograd_sync_streams (__main__.TensorPipeTensorPipeCudaDistAutogradTest)
2022-05-18T03:43:37.3718327Z test_gradients_synchronizations (__main__.TensorPipeTensorPipeCudaDistAutogradTest)
2022-05-18T03:43:37.9206795Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv1lv8kej
2022-05-18T03:43:37.9207740Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv1lv8kej/_remote_module_non_scriptable.py
2022-05-18T03:43:38.1685093Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T03:43:38.1693989Z 
2022-05-18T03:43:38.1694132Z Running tests...
2022-05-18T03:43:38.1694774Z ----------------------------------------------------------------------
2022-05-18T03:43:38.1716992Z   test_ddp_dist_autograd_local_vs_remote_gpu (__main__.TensorPipeCudaDdpComparisonTest) ... skip: c10d was not compiled with the NCCL backend (0.002s)
2022-05-18T03:43:38.1717369Z 
2022-05-18T03:43:38.1718028Z ----------------------------------------------------------------------
2022-05-18T03:43:38.1718301Z Ran 1 test in 0.002s
2022-05-18T03:43:38.1718421Z 
2022-05-18T03:43:38.1718500Z OK (skipped=1)
2022-05-18T03:43:38.1718635Z 
2022-05-18T03:43:38.1718755Z Generating XML reports...
2022-05-18T03:43:38.1748575Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaDdpComparisonTest-20220518034338.xml
2022-05-18T03:43:38.8240770Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpppsb0j4n
2022-05-18T03:43:38.8241477Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpppsb0j4n/_remote_module_non_scriptable.py
2022-05-18T03:43:39.0705722Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T03:43:39.0714860Z 
2022-05-18T03:43:39.0714987Z Running tests...
2022-05-18T03:43:39.0715568Z ----------------------------------------------------------------------
2022-05-18T03:43:39.3818000Z   test_gpu_simple (__main__.TensorPipeCudaDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28862
2022-05-18T03:43:39.3840860Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28863
2022-05-18T03:43:39.3863620Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28864
2022-05-18T03:43:39.3887576Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 28865
2022-05-18T03:43:40.0597860Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpoa_hzk54
2022-05-18T03:43:40.0598618Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpoa_hzk54/_remote_module_non_scriptable.py
2022-05-18T03:43:40.0662822Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzmmth9gt
2022-05-18T03:43:40.0664195Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzmmth9gt/_remote_module_non_scriptable.py
2022-05-18T03:43:40.0926579Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkijghudg
2022-05-18T03:43:40.0927551Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkijghudg/_remote_module_non_scriptable.py
2022-05-18T03:43:40.1418866Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6mi0i7p_
2022-05-18T03:43:40.1419670Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6mi0i7p_/_remote_module_non_scriptable.py
2022-05-18T03:43:40.3098070Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:43:40.3179608Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:43:40.3429109Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:43:40.4205823Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:43:40.5923811Z skip: Need at least 1 CUDA device (1.521s)
2022-05-18T03:43:40.5924064Z 
2022-05-18T03:43:40.5924402Z ----------------------------------------------------------------------
2022-05-18T03:43:40.5924674Z Ran 1 test in 1.521s
2022-05-18T03:43:40.5924806Z 
2022-05-18T03:43:40.5924882Z OK (skipped=1)
2022-05-18T03:43:40.5924990Z 
2022-05-18T03:43:40.5925076Z Generating XML reports...
2022-05-18T03:43:40.5957213Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaDistAutogradTest-20220518034339.xml
2022-05-18T03:43:41.3332167Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplgat8_ky
2022-05-18T03:43:41.3332979Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplgat8_ky/_remote_module_non_scriptable.py
2022-05-18T03:43:41.5814497Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T03:43:41.5824122Z 
2022-05-18T03:43:41.5824250Z Running tests...
2022-05-18T03:43:41.5824592Z ----------------------------------------------------------------------
2022-05-18T03:43:41.8925660Z   test_gpu_to_cpu_continuation (__main__.TensorPipeCudaDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28917
2022-05-18T03:43:41.8948317Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28918
2022-05-18T03:43:41.8970747Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28919
2022-05-18T03:43:41.8994350Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 28920
2022-05-18T03:43:42.4717466Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp35s4aw4i
2022-05-18T03:43:42.4718177Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkggn6bi0
2022-05-18T03:43:42.4718808Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp35s4aw4i/_remote_module_non_scriptable.py
2022-05-18T03:43:42.4720311Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkggn6bi0/_remote_module_non_scriptable.py
2022-05-18T03:43:42.5197237Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4v_odqie
2022-05-18T03:43:42.5198830Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4v_odqie/_remote_module_non_scriptable.py
2022-05-18T03:43:42.5274409Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1bp9cjrj
2022-05-18T03:43:42.5275887Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1bp9cjrj/_remote_module_non_scriptable.py
2022-05-18T03:43:42.7182646Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:43:42.7207489Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:43:42.7694763Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:43:42.7775271Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:43:43.0028510Z skip: Need at least 1 CUDA device (1.420s)
2022-05-18T03:43:43.0029082Z 
2022-05-18T03:43:43.0029615Z ----------------------------------------------------------------------
2022-05-18T03:43:43.0030091Z Ran 1 test in 1.420s
2022-05-18T03:43:43.0030285Z 
2022-05-18T03:43:43.0030431Z OK (skipped=1)
2022-05-18T03:43:43.0030636Z 
2022-05-18T03:43:43.0030799Z Generating XML reports...
2022-05-18T03:43:43.0061189Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaDistAutogradTest-20220518034341.xml
2022-05-18T03:43:43.7434028Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp10p1uzl6
2022-05-18T03:43:43.7434789Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp10p1uzl6/_remote_module_non_scriptable.py
2022-05-18T03:43:43.9901986Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T03:43:43.9911941Z 
2022-05-18T03:43:43.9912038Z Running tests...
2022-05-18T03:43:43.9912952Z ----------------------------------------------------------------------
2022-05-18T03:43:44.3017314Z   test_gpu_to_cpu_continuation_gpu_root (__main__.TensorPipeCudaDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28972
2022-05-18T03:43:44.3040052Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28973
2022-05-18T03:43:44.3062700Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28974
2022-05-18T03:43:44.3086648Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 28975
2022-05-18T03:43:44.9425673Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnl1tp9ug
2022-05-18T03:43:44.9426662Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnl1tp9ug/_remote_module_non_scriptable.py
2022-05-18T03:43:44.9432340Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqzo8vey2
2022-05-18T03:43:44.9434746Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqzo8vey2/_remote_module_non_scriptable.py
2022-05-18T03:43:44.9552219Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg7wd2_1b
2022-05-18T03:43:44.9553777Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg7wd2_1b/_remote_module_non_scriptable.py
2022-05-18T03:43:44.9658294Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfe_9y3gs
2022-05-18T03:43:44.9659739Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfe_9y3gs/_remote_module_non_scriptable.py
2022-05-18T03:43:45.1908179Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:43:45.1918276Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:43:45.2034265Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:43:45.2186635Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:43:45.4121160Z skip: Need at least 1 CUDA device (1.421s)
2022-05-18T03:43:45.4121565Z 
2022-05-18T03:43:45.4122077Z ----------------------------------------------------------------------
2022-05-18T03:43:45.4122522Z Ran 1 test in 1.421s
2022-05-18T03:43:45.4122730Z 
2022-05-18T03:43:45.4122864Z OK (skipped=1)
2022-05-18T03:43:45.4123149Z 
2022-05-18T03:43:45.4123290Z Generating XML reports...
2022-05-18T03:43:45.4154430Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaDistAutogradTest-20220518034343.xml
2022-05-18T03:43:46.1651846Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi_lsbbi8
2022-05-18T03:43:46.1652290Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi_lsbbi8/_remote_module_non_scriptable.py
2022-05-18T03:43:46.4114013Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T03:43:46.4123723Z 
2022-05-18T03:43:46.4123877Z Running tests...
2022-05-18T03:43:46.4124263Z ----------------------------------------------------------------------
2022-05-18T03:43:46.7253921Z   test_input_moved_to_cuda_device (__main__.TensorPipeCudaRemoteModuleTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29027
2022-05-18T03:43:46.7276095Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29028
2022-05-18T03:43:47.2851982Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpinfkhrh5
2022-05-18T03:43:47.2852737Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpinfkhrh5/_remote_module_non_scriptable.py
2022-05-18T03:43:47.3071853Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprcsdupfn
2022-05-18T03:43:47.3073610Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprcsdupfn/_remote_module_non_scriptable.py
2022-05-18T03:43:47.5337639Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:43:47.5537244Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:43:47.7302780Z skip: Need at least 1 CUDA device (1.318s)
2022-05-18T03:43:47.7303305Z 
2022-05-18T03:43:47.7303794Z ----------------------------------------------------------------------
2022-05-18T03:43:47.7304340Z Ran 1 test in 1.318s
2022-05-18T03:43:47.7304536Z 
2022-05-18T03:43:47.7304628Z OK (skipped=1)
2022-05-18T03:43:47.7304738Z 
2022-05-18T03:43:47.7304812Z Generating XML reports...
2022-05-18T03:43:47.7336426Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaRemoteModuleTest-20220518034346.xml
2022-05-18T03:43:48.4728471Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzeuisuj3
2022-05-18T03:43:48.4729465Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzeuisuj3/_remote_module_non_scriptable.py
2022-05-18T03:43:48.7194287Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T03:43:48.7204401Z 
2022-05-18T03:43:48.7204761Z Running tests...
2022-05-18T03:43:48.7205388Z ----------------------------------------------------------------------
2022-05-18T03:43:49.0324824Z   test_input_moved_to_cuda_device_script (__main__.TensorPipeCudaRemoteModuleTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29060
2022-05-18T03:43:49.0347530Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29061
2022-05-18T03:43:49.5865204Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi5rwn6h1
2022-05-18T03:43:49.5865736Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphuoo3mwd
2022-05-18T03:43:49.5866431Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi5rwn6h1/_remote_module_non_scriptable.py
2022-05-18T03:43:49.5866851Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphuoo3mwd/_remote_module_non_scriptable.py
2022-05-18T03:43:49.8297146Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:43:49.8329513Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:43:50.0375977Z skip: Need at least 1 CUDA device (1.317s)
2022-05-18T03:43:50.0376302Z 
2022-05-18T03:43:50.0376669Z ----------------------------------------------------------------------
2022-05-18T03:43:50.0376944Z Ran 1 test in 1.317s
2022-05-18T03:43:50.0377059Z 
2022-05-18T03:43:50.0377121Z OK (skipped=1)
2022-05-18T03:43:50.0377227Z 
2022-05-18T03:43:50.0377313Z Generating XML reports...
2022-05-18T03:43:50.0409541Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaRemoteModuleTest-20220518034348.xml
2022-05-18T03:43:50.7891792Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0iy7huhm
2022-05-18T03:43:50.7892627Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0iy7huhm/_remote_module_non_scriptable.py
2022-05-18T03:43:51.0375605Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T03:43:51.0384967Z 
2022-05-18T03:43:51.0385078Z Running tests...
2022-05-18T03:43:51.0386010Z ----------------------------------------------------------------------
2022-05-18T03:43:51.3538174Z   test_invalid_devices (__main__.TensorPipeCudaRemoteModuleTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29093
2022-05-18T03:43:51.3561431Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29094
2022-05-18T03:43:51.9151818Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcm8l_e2d
2022-05-18T03:43:51.9152793Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcm8l_e2d/_remote_module_non_scriptable.py
2022-05-18T03:43:51.9435265Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp06vbmqo9
2022-05-18T03:43:51.9436190Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp06vbmqo9/_remote_module_non_scriptable.py
2022-05-18T03:43:52.1595420Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:43:52.1904010Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:43:52.3589140Z skip: Need at least 1 CUDA device (1.320s)
2022-05-18T03:43:52.3589369Z 
2022-05-18T03:43:52.3589676Z ----------------------------------------------------------------------
2022-05-18T03:43:52.3590001Z Ran 1 test in 1.320s
2022-05-18T03:43:52.3590114Z 
2022-05-18T03:43:52.3590188Z OK (skipped=1)
2022-05-18T03:43:52.3590295Z 
2022-05-18T03:43:52.3590381Z Generating XML reports...
2022-05-18T03:43:52.3621649Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaRemoteModuleTest-20220518034351.xml
2022-05-18T03:43:53.1014301Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyxrqzqhi
2022-05-18T03:43:53.1015107Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyxrqzqhi/_remote_module_non_scriptable.py
2022-05-18T03:43:53.3482169Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T03:43:53.3491221Z 
2022-05-18T03:43:53.3491365Z Running tests...
2022-05-18T03:43:53.3491800Z ----------------------------------------------------------------------
2022-05-18T03:43:53.6648153Z   test_valid_device (__main__.TensorPipeCudaRemoteModuleTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29126
2022-05-18T03:43:53.6670934Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29127
2022-05-18T03:43:54.2162794Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvqpkhed8
2022-05-18T03:43:54.2163542Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvqpkhed8/_remote_module_non_scriptable.py
2022-05-18T03:43:54.2164210Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr0d7a7y9
2022-05-18T03:43:54.2166883Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr0d7a7y9/_remote_module_non_scriptable.py
2022-05-18T03:43:54.4605995Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:43:54.4619804Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:43:54.6698062Z skip: Need at least 1 CUDA device (1.320s)
2022-05-18T03:43:54.6698374Z 
2022-05-18T03:43:54.6698868Z ----------------------------------------------------------------------
2022-05-18T03:43:54.6699139Z Ran 1 test in 1.321s
2022-05-18T03:43:54.6699257Z 
2022-05-18T03:43:54.6699333Z OK (skipped=1)
2022-05-18T03:43:54.6699426Z 
2022-05-18T03:43:54.6699513Z Generating XML reports...
2022-05-18T03:43:54.6731192Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaRemoteModuleTest-20220518034353.xml
2022-05-18T03:43:55.4193542Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp219ars80
2022-05-18T03:43:55.4194187Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp219ars80/_remote_module_non_scriptable.py
2022-05-18T03:43:55.6678580Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T03:43:55.6688419Z 
2022-05-18T03:43:55.6688525Z Running tests...
2022-05-18T03:43:55.6689104Z ----------------------------------------------------------------------
2022-05-18T03:43:55.9828843Z   test_profiler_remote_cuda (__main__.TensorPipeCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29159
2022-05-18T03:43:55.9850951Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29160
2022-05-18T03:43:55.9873539Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29161
2022-05-18T03:43:55.9896817Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 29162
2022-05-18T03:43:56.5666399Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp84pu_lbe
2022-05-18T03:43:56.5667287Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp84pu_lbe/_remote_module_non_scriptable.py
2022-05-18T03:43:56.6085145Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_x3ub5uj
2022-05-18T03:43:56.6086131Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_x3ub5uj/_remote_module_non_scriptable.py
2022-05-18T03:43:56.6140060Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkj25ff0s
2022-05-18T03:43:56.6141701Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkj25ff0s/_remote_module_non_scriptable.py
2022-05-18T03:43:56.6194421Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwt94enc6
2022-05-18T03:43:56.6196375Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwt94enc6/_remote_module_non_scriptable.py
2022-05-18T03:43:56.8151471Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:43:56.8580844Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:43:56.8631684Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:43:56.8650595Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:43:57.0931901Z skip: Need at least 2 CUDA devices (1.424s)
2022-05-18T03:43:57.0932230Z 
2022-05-18T03:43:57.0932534Z ----------------------------------------------------------------------
2022-05-18T03:43:57.0932784Z Ran 1 test in 1.424s
2022-05-18T03:43:57.0932899Z 
2022-05-18T03:43:57.0932974Z OK (skipped=1)
2022-05-18T03:43:57.0933084Z 
2022-05-18T03:43:57.0933158Z Generating XML reports...
2022-05-18T03:43:57.0964823Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaRpcTest-20220518034355.xml
2022-05-18T03:43:57.8371231Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsb8ed48b
2022-05-18T03:43:57.8371850Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsb8ed48b/_remote_module_non_scriptable.py
2022-05-18T03:43:58.0847068Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T03:43:58.0856291Z 
2022-05-18T03:43:58.0856372Z Running tests...
2022-05-18T03:43:58.0856819Z ----------------------------------------------------------------------
2022-05-18T03:43:58.3958853Z   test_basic_gloo_ckpt_always (__main__.TensorPipePipeWithDDPTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29214
2022-05-18T03:43:58.3979918Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29215
2022-05-18T03:43:58.9482410Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4tc0kyjl
2022-05-18T03:43:58.9483114Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4tc0kyjl/_remote_module_non_scriptable.py
2022-05-18T03:43:58.9493916Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph5a47wys
2022-05-18T03:43:58.9496057Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph5a47wys/_remote_module_non_scriptable.py
2022-05-18T03:43:59.1927354Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:43:59.1942114Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:43:59.4006627Z skip: Need at least 4 CUDA devices (1.315s)
2022-05-18T03:43:59.4006903Z 
2022-05-18T03:43:59.4007257Z ----------------------------------------------------------------------
2022-05-18T03:43:59.4007551Z Ran 1 test in 1.315s
2022-05-18T03:43:59.4007665Z 
2022-05-18T03:43:59.4007740Z OK (skipped=1)
2022-05-18T03:43:59.4007846Z 
2022-05-18T03:43:59.4007958Z Generating XML reports...
2022-05-18T03:43:59.4040129Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20220518034358.xml
2022-05-18T03:44:00.1495197Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5sy5g2vy
2022-05-18T03:44:00.1495882Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5sy5g2vy/_remote_module_non_scriptable.py
2022-05-18T03:44:00.3976232Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T03:44:00.3985191Z 
2022-05-18T03:44:00.3985438Z Running tests...
2022-05-18T03:44:00.3986046Z ----------------------------------------------------------------------
2022-05-18T03:44:00.7091793Z   test_basic_gloo_ckpt_except_last (__main__.TensorPipePipeWithDDPTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29247
2022-05-18T03:44:00.7113979Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29248
2022-05-18T03:44:01.2646494Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpccvk70bl
2022-05-18T03:44:01.2647476Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpccvk70bl/_remote_module_non_scriptable.py
2022-05-18T03:44:01.2894930Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9dmod2y4
2022-05-18T03:44:01.2896298Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9dmod2y4/_remote_module_non_scriptable.py
2022-05-18T03:44:01.5159450Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:44:01.5366362Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:44:01.7141288Z skip: Need at least 4 CUDA devices (1.315s)
2022-05-18T03:44:01.7141617Z 
2022-05-18T03:44:01.7142080Z ----------------------------------------------------------------------
2022-05-18T03:44:01.7142335Z Ran 1 test in 1.315s
2022-05-18T03:44:01.7142699Z 
2022-05-18T03:44:01.7142760Z OK (skipped=1)
2022-05-18T03:44:01.7143014Z 
2022-05-18T03:44:01.7143104Z Generating XML reports...
2022-05-18T03:44:01.7173983Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20220518034400.xml
2022-05-18T03:44:02.4679638Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4q6qbprz
2022-05-18T03:44:02.4680373Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4q6qbprz/_remote_module_non_scriptable.py
2022-05-18T03:44:02.7158851Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T03:44:02.7168924Z 
2022-05-18T03:44:02.7169231Z Running tests...
2022-05-18T03:44:02.7169846Z ----------------------------------------------------------------------
2022-05-18T03:44:03.0303781Z   test_basic_gloo_ckpt_never (__main__.TensorPipePipeWithDDPTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29280
2022-05-18T03:44:03.0325300Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29281
2022-05-18T03:44:03.5982551Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9iuzqju7
2022-05-18T03:44:03.5983247Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9iuzqju7/_remote_module_non_scriptable.py
2022-05-18T03:44:03.6087763Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjqszqglp
2022-05-18T03:44:03.6089281Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjqszqglp/_remote_module_non_scriptable.py
2022-05-18T03:44:03.8436094Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:44:03.8524117Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:44:04.0352655Z skip: Need at least 4 CUDA devices (1.318s)
2022-05-18T03:44:04.0352959Z 
2022-05-18T03:44:04.0353491Z ----------------------------------------------------------------------
2022-05-18T03:44:04.0353802Z Ran 1 test in 1.318s
2022-05-18T03:44:04.0353918Z 
2022-05-18T03:44:04.0353978Z OK (skipped=1)
2022-05-18T03:44:04.0354096Z 
2022-05-18T03:44:04.0354183Z Generating XML reports...
2022-05-18T03:44:04.0386113Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20220518034402.xml
2022-05-18T03:44:04.7845667Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptgcdrnqv
2022-05-18T03:44:04.7846341Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptgcdrnqv/_remote_module_non_scriptable.py
2022-05-18T03:44:05.0315924Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T03:44:05.0325619Z 
2022-05-18T03:44:05.0325826Z Running tests...
2022-05-18T03:44:05.0326277Z ----------------------------------------------------------------------
2022-05-18T03:44:05.3421560Z   test_basic_gloo_ckpt_never_find_unused (__main__.TensorPipePipeWithDDPTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29313
2022-05-18T03:44:05.3444073Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29314
2022-05-18T03:44:05.9068355Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjis5x6jz
2022-05-18T03:44:05.9069092Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx8kl188r
2022-05-18T03:44:05.9069712Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjis5x6jz/_remote_module_non_scriptable.py
2022-05-18T03:44:05.9070454Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx8kl188r/_remote_module_non_scriptable.py
2022-05-18T03:44:06.1514687Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:44:06.1516950Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:44:06.3471606Z skip: Need at least 4 CUDA devices (1.314s)
2022-05-18T03:44:06.3483691Z 
2022-05-18T03:44:06.3484310Z ----------------------------------------------------------------------
2022-05-18T03:44:06.3484577Z Ran 1 test in 1.315s
2022-05-18T03:44:06.3484687Z 
2022-05-18T03:44:06.3484751Z OK (skipped=1)
2022-05-18T03:44:06.3484854Z 
2022-05-18T03:44:06.3484931Z Generating XML reports...
2022-05-18T03:44:06.3505558Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20220518034405.xml
2022-05-18T03:44:07.0901697Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_1wy035w
2022-05-18T03:44:07.0902348Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_1wy035w/_remote_module_non_scriptable.py
2022-05-18T03:44:07.3386605Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T03:44:07.3395852Z 
2022-05-18T03:44:07.3395963Z Running tests...
2022-05-18T03:44:07.3396743Z ----------------------------------------------------------------------
2022-05-18T03:44:07.3401686Z   test_basic_nccl_ckpt_always (__main__.TensorPipePipeWithDDPTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-05-18T03:44:07.3402017Z 
2022-05-18T03:44:07.3402431Z ----------------------------------------------------------------------
2022-05-18T03:44:07.3402891Z Ran 1 test in 0.001s
2022-05-18T03:44:07.3403043Z 
2022-05-18T03:44:07.3403117Z OK (skipped=1)
2022-05-18T03:44:07.3403225Z 
2022-05-18T03:44:07.3403316Z Generating XML reports...
2022-05-18T03:44:07.3433194Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20220518034407.xml
2022-05-18T03:44:07.9956248Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4icl_f5n
2022-05-18T03:44:07.9956692Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4icl_f5n/_remote_module_non_scriptable.py
2022-05-18T03:44:08.2407231Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T03:44:08.2416792Z 
2022-05-18T03:44:08.2416915Z Running tests...
2022-05-18T03:44:08.2417509Z ----------------------------------------------------------------------
2022-05-18T03:44:08.2422237Z   test_basic_nccl_ckpt_except_last (__main__.TensorPipePipeWithDDPTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-05-18T03:44:08.2422563Z 
2022-05-18T03:44:08.2423550Z ----------------------------------------------------------------------
2022-05-18T03:44:08.2423966Z Ran 1 test in 0.001s
2022-05-18T03:44:08.2424086Z 
2022-05-18T03:44:08.2424166Z OK (skipped=1)
2022-05-18T03:44:08.2424282Z 
2022-05-18T03:44:08.2424371Z Generating XML reports...
2022-05-18T03:44:08.2454219Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20220518034408.xml
2022-05-18T03:44:08.8944956Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1z7pyask
2022-05-18T03:44:08.8945866Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1z7pyask/_remote_module_non_scriptable.py
2022-05-18T03:44:09.1414565Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T03:44:09.1424228Z 
2022-05-18T03:44:09.1424368Z Running tests...
2022-05-18T03:44:09.1425102Z ----------------------------------------------------------------------
2022-05-18T03:44:09.1429385Z   test_basic_nccl_ckpt_never (__main__.TensorPipePipeWithDDPTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-05-18T03:44:09.1429742Z 
2022-05-18T03:44:09.1430003Z ----------------------------------------------------------------------
2022-05-18T03:44:09.1430249Z Ran 1 test in 0.001s
2022-05-18T03:44:09.1430365Z 
2022-05-18T03:44:09.1430475Z OK (skipped=1)
2022-05-18T03:44:09.1430583Z 
2022-05-18T03:44:09.1430889Z Generating XML reports...
2022-05-18T03:44:09.1460305Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20220518034409.xml
2022-05-18T03:44:09.7962167Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdpjk21bv
2022-05-18T03:44:09.7962690Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdpjk21bv/_remote_module_non_scriptable.py
2022-05-18T03:44:10.0447763Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T03:44:10.0457572Z 
2022-05-18T03:44:10.0457691Z Running tests...
2022-05-18T03:44:10.0458118Z ----------------------------------------------------------------------
2022-05-18T03:44:10.0463336Z   test_basic_nccl_ckpt_never_find_unused (__main__.TensorPipePipeWithDDPTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-05-18T03:44:10.0463855Z 
2022-05-18T03:44:10.0464275Z ----------------------------------------------------------------------
2022-05-18T03:44:10.0464650Z Ran 1 test in 0.001s
2022-05-18T03:44:10.0464764Z 
2022-05-18T03:44:10.0464838Z OK (skipped=1)
2022-05-18T03:44:10.0464947Z 
2022-05-18T03:44:10.0465041Z Generating XML reports...
2022-05-18T03:44:10.0495361Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20220518034410.xml
2022-05-18T03:44:10.7077203Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_w14go32
2022-05-18T03:44:10.7077947Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_w14go32/_remote_module_non_scriptable.py
2022-05-18T03:44:10.9566678Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T03:44:10.9576799Z 
2022-05-18T03:44:10.9577246Z Running tests...
2022-05-18T03:44:10.9577665Z ----------------------------------------------------------------------
2022-05-18T03:44:11.2727339Z   test_async_execution_nested_with_cuda_future (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29386
2022-05-18T03:44:11.2749524Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29387
2022-05-18T03:44:11.2772272Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29388
2022-05-18T03:44:11.2796226Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 29389
2022-05-18T03:44:11.8536721Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpahojsr54
2022-05-18T03:44:11.8537632Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpahojsr54/_remote_module_non_scriptable.py
2022-05-18T03:44:11.8786029Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbppk6ruw
2022-05-18T03:44:11.8786756Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbppk6ruw/_remote_module_non_scriptable.py
2022-05-18T03:44:11.8959725Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4g75vmvo
2022-05-18T03:44:11.8960664Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4g75vmvo/_remote_module_non_scriptable.py
2022-05-18T03:44:11.9250983Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpoq_abbjx
2022-05-18T03:44:11.9252166Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpoq_abbjx/_remote_module_non_scriptable.py
2022-05-18T03:44:12.1031013Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:44:12.1287737Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:44:12.1466542Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:44:12.1888227Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:44:12.3829926Z skip: Need at least 1 CUDA device (1.425s)
2022-05-18T03:44:12.3830576Z 
2022-05-18T03:44:12.3830883Z ----------------------------------------------------------------------
2022-05-18T03:44:12.3831132Z Ran 1 test in 1.425s
2022-05-18T03:44:12.3831247Z 
2022-05-18T03:44:12.3831400Z OK (skipped=1)
2022-05-18T03:44:12.3831509Z 
2022-05-18T03:44:12.3831583Z Generating XML reports...
2022-05-18T03:44:12.3864215Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518034410.xml
2022-05-18T03:44:13.1431716Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5wbkme9k
2022-05-18T03:44:13.1432461Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5wbkme9k/_remote_module_non_scriptable.py
2022-05-18T03:44:13.3982456Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T03:44:13.3992882Z 
2022-05-18T03:44:13.3992972Z Running tests...
2022-05-18T03:44:13.3994118Z ----------------------------------------------------------------------
2022-05-18T03:44:13.7283274Z   test_async_execution_with_cuda_future (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29441
2022-05-18T03:44:13.7305007Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29442
2022-05-18T03:44:13.7328396Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29443
2022-05-18T03:44:13.7352626Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 29444
2022-05-18T03:44:14.4033780Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdc4qg6f5
2022-05-18T03:44:14.4034419Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdc4qg6f5/_remote_module_non_scriptable.py
2022-05-18T03:44:14.4504466Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk1r06bef
2022-05-18T03:44:14.4505739Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk1r06bef/_remote_module_non_scriptable.py
2022-05-18T03:44:14.5481574Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8ad1ajab
2022-05-18T03:44:14.5483197Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8ad1ajab/_remote_module_non_scriptable.py
2022-05-18T03:44:14.5605917Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3ko9pplz
2022-05-18T03:44:14.5608101Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3ko9pplz/_remote_module_non_scriptable.py
2022-05-18T03:44:14.6532085Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:44:14.7004068Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:44:14.8105608Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:44:14.8202837Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:44:15.0391737Z skip: Need at least 1 CUDA device (1.640s)
2022-05-18T03:44:15.0392043Z 
2022-05-18T03:44:15.0392418Z ----------------------------------------------------------------------
2022-05-18T03:44:15.0392676Z Ran 1 test in 1.640s
2022-05-18T03:44:15.0392791Z 
2022-05-18T03:44:15.0392864Z OK (skipped=1)
2022-05-18T03:44:15.0392958Z 
2022-05-18T03:44:15.0393046Z Generating XML reports...
2022-05-18T03:44:15.0424596Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518034413.xml
2022-05-18T03:44:15.7943794Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6qhxi4sb
2022-05-18T03:44:15.7944521Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6qhxi4sb/_remote_module_non_scriptable.py
2022-05-18T03:44:16.0407135Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T03:44:16.0417152Z 
2022-05-18T03:44:16.0417557Z Running tests...
2022-05-18T03:44:16.0417961Z ----------------------------------------------------------------------
2022-05-18T03:44:16.3558948Z   test_cuda_future_callback_changes_devices (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29496
2022-05-18T03:44:16.3580819Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29497
2022-05-18T03:44:16.3604027Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29498
2022-05-18T03:44:16.3628006Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 29499
2022-05-18T03:44:17.0124435Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdxrlevyc
2022-05-18T03:44:17.0125642Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdxrlevyc/_remote_module_non_scriptable.py
2022-05-18T03:44:17.0207611Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpihi1y_7s
2022-05-18T03:44:17.0209027Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpihi1y_7s/_remote_module_non_scriptable.py
2022-05-18T03:44:17.0233033Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf7qc_p8w
2022-05-18T03:44:17.0234591Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf7qc_p8w/_remote_module_non_scriptable.py
2022-05-18T03:44:17.0467191Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfugar6ny
2022-05-18T03:44:17.0468160Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfugar6ny/_remote_module_non_scriptable.py
2022-05-18T03:44:17.2581265Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:44:17.2677044Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:44:17.2754439Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:44:17.2961653Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:44:17.4663472Z skip: Need at least 2 CUDA devices (1.424s)
2022-05-18T03:44:17.4664004Z 
2022-05-18T03:44:17.4664641Z ----------------------------------------------------------------------
2022-05-18T03:44:17.4665096Z Ran 1 test in 1.425s
2022-05-18T03:44:17.4665294Z 
2022-05-18T03:44:17.4665373Z OK (skipped=1)
2022-05-18T03:44:17.4665482Z 
2022-05-18T03:44:17.4665570Z Generating XML reports...
2022-05-18T03:44:17.4696368Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518034416.xml
2022-05-18T03:44:18.2096571Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvduaioln
2022-05-18T03:44:18.2097229Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvduaioln/_remote_module_non_scriptable.py
2022-05-18T03:44:18.4558323Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T03:44:18.4568051Z 
2022-05-18T03:44:18.4568365Z Running tests...
2022-05-18T03:44:18.4569037Z ----------------------------------------------------------------------
2022-05-18T03:44:18.7665058Z   test_cuda_future_can_extract_cuda_sparse_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29551
2022-05-18T03:44:18.7687426Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29552
2022-05-18T03:44:18.7710451Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29553
2022-05-18T03:44:18.7733606Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 29554
2022-05-18T03:44:19.3922141Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph3w8ddnj
2022-05-18T03:44:19.3922918Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph3w8ddnj/_remote_module_non_scriptable.py
2022-05-18T03:44:19.3984479Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8nhanal5
2022-05-18T03:44:19.3986551Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8nhanal5/_remote_module_non_scriptable.py
2022-05-18T03:44:19.4139503Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpton3o3qw
2022-05-18T03:44:19.4141005Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpton3o3qw/_remote_module_non_scriptable.py
2022-05-18T03:44:19.4224464Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnoofjztp
2022-05-18T03:44:19.4225963Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnoofjztp/_remote_module_non_scriptable.py
2022-05-18T03:44:19.6424917Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:44:19.6450375Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:44:19.6648498Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:44:19.6719817Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:44:19.8768938Z skip: Need at least 1 CUDA device (1.420s)
2022-05-18T03:44:19.8769253Z 
2022-05-18T03:44:19.8769779Z ----------------------------------------------------------------------
2022-05-18T03:44:19.8770143Z Ran 1 test in 1.420s
2022-05-18T03:44:19.8770262Z 
2022-05-18T03:44:19.8770322Z OK (skipped=1)
2022-05-18T03:44:19.8770432Z 
2022-05-18T03:44:19.8770521Z Generating XML reports...
2022-05-18T03:44:19.8802313Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518034418.xml
2022-05-18T03:44:20.6228853Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz27tt9xi
2022-05-18T03:44:20.6229322Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz27tt9xi/_remote_module_non_scriptable.py
2022-05-18T03:44:20.8694154Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T03:44:20.8704241Z 
2022-05-18T03:44:20.8704723Z Running tests...
2022-05-18T03:44:20.8705152Z ----------------------------------------------------------------------
2022-05-18T03:44:21.1831643Z   test_cuda_future_can_extract_cuda_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29606
2022-05-18T03:44:21.1854555Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29607
2022-05-18T03:44:21.1877565Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29608
2022-05-18T03:44:21.1901049Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 29609
2022-05-18T03:44:21.8046701Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn6a9ra0o
2022-05-18T03:44:21.8047511Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn6a9ra0o/_remote_module_non_scriptable.py
2022-05-18T03:44:21.8136884Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3pttq1ro
2022-05-18T03:44:21.8138579Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3pttq1ro/_remote_module_non_scriptable.py
2022-05-18T03:44:21.8166143Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp30j3ganj
2022-05-18T03:44:21.8167553Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp30j3ganj/_remote_module_non_scriptable.py
2022-05-18T03:44:21.8240433Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpohlzvkkc
2022-05-18T03:44:21.8242337Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpohlzvkkc/_remote_module_non_scriptable.py
2022-05-18T03:44:22.0505706Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:44:22.0602619Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:44:22.0660444Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:44:22.0791080Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:44:22.2936834Z skip: Need at least 1 CUDA device (1.423s)
2022-05-18T03:44:22.2937130Z 
2022-05-18T03:44:22.2937658Z ----------------------------------------------------------------------
2022-05-18T03:44:22.2937940Z Ran 1 test in 1.423s
2022-05-18T03:44:22.2938054Z 
2022-05-18T03:44:22.2938116Z OK (skipped=1)
2022-05-18T03:44:22.2938228Z 
2022-05-18T03:44:22.2938315Z Generating XML reports...
2022-05-18T03:44:22.2969691Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518034420.xml
2022-05-18T03:44:23.0430289Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnjrde9vy
2022-05-18T03:44:23.0431482Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnjrde9vy/_remote_module_non_scriptable.py
2022-05-18T03:44:23.2899608Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T03:44:23.2909600Z 
2022-05-18T03:44:23.2909713Z Running tests...
2022-05-18T03:44:23.2910117Z ----------------------------------------------------------------------
2022-05-18T03:44:23.6027843Z   test_cuda_future_can_extract_custom_class_with_cuda_sparse_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29661
2022-05-18T03:44:23.6050020Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29662
2022-05-18T03:44:23.6072528Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29663
2022-05-18T03:44:23.6096045Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 29664
2022-05-18T03:44:24.2316565Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt31af72v
2022-05-18T03:44:24.2317332Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt31af72v/_remote_module_non_scriptable.py
2022-05-18T03:44:24.2373538Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp86c5qaq
2022-05-18T03:44:24.2375072Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp86c5qaq/_remote_module_non_scriptable.py
2022-05-18T03:44:24.2450395Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7qfwwb43
2022-05-18T03:44:24.2452862Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7qfwwb43/_remote_module_non_scriptable.py
2022-05-18T03:44:24.2460727Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnkmrmjld
2022-05-18T03:44:24.2463109Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnkmrmjld/_remote_module_non_scriptable.py
2022-05-18T03:44:24.4816691Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:44:24.4833042Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:44:24.4965295Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:44:24.4987059Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:44:24.7129834Z skip: Need at least 1 CUDA device (1.422s)
2022-05-18T03:44:24.7130153Z 
2022-05-18T03:44:24.7130574Z ----------------------------------------------------------------------
2022-05-18T03:44:24.7130861Z Ran 1 test in 1.422s
2022-05-18T03:44:24.7131032Z 
2022-05-18T03:44:24.7131107Z OK (skipped=1)
2022-05-18T03:44:24.7131217Z 
2022-05-18T03:44:24.7131307Z Generating XML reports...
2022-05-18T03:44:24.7164988Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518034423.xml
2022-05-18T03:44:25.4666613Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphauyzzr7
2022-05-18T03:44:25.4667269Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphauyzzr7/_remote_module_non_scriptable.py
2022-05-18T03:44:25.7138462Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T03:44:25.7148189Z 
2022-05-18T03:44:25.7148386Z Running tests...
2022-05-18T03:44:25.7148738Z ----------------------------------------------------------------------
2022-05-18T03:44:26.0287933Z   test_cuda_future_can_extract_custom_class_with_cuda_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29716
2022-05-18T03:44:26.0310100Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29717
2022-05-18T03:44:26.0333565Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29718
2022-05-18T03:44:26.0357306Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 29719
2022-05-18T03:44:26.6175832Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw16b2ji6
2022-05-18T03:44:26.6176606Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw16b2ji6/_remote_module_non_scriptable.py
2022-05-18T03:44:26.6566577Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5i7t6mn7
2022-05-18T03:44:26.6567348Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5i7t6mn7/_remote_module_non_scriptable.py
2022-05-18T03:44:26.6637026Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1pguqch4
2022-05-18T03:44:26.6639325Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1pguqch4/_remote_module_non_scriptable.py
2022-05-18T03:44:26.6815443Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxkh_6fkl
2022-05-18T03:44:26.6816411Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxkh_6fkl/_remote_module_non_scriptable.py
2022-05-18T03:44:26.8662589Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:44:26.9055290Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:44:26.9129953Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:44:26.9340491Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:44:27.1391861Z skip: Need at least 1 CUDA device (1.424s)
2022-05-18T03:44:27.1392203Z 
2022-05-18T03:44:27.1392700Z ----------------------------------------------------------------------
2022-05-18T03:44:27.1392961Z Ran 1 test in 1.424s
2022-05-18T03:44:27.1393076Z 
2022-05-18T03:44:27.1393150Z OK (skipped=1)
2022-05-18T03:44:27.1393259Z 
2022-05-18T03:44:27.1393346Z Generating XML reports...
2022-05-18T03:44:27.1424484Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518034425.xml
2022-05-18T03:44:27.8952439Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp2_505ny
2022-05-18T03:44:27.8953191Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp2_505ny/_remote_module_non_scriptable.py
2022-05-18T03:44:28.1422343Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T03:44:28.1432282Z 
2022-05-18T03:44:28.1432772Z Running tests...
2022-05-18T03:44:28.1433192Z ----------------------------------------------------------------------
2022-05-18T03:44:28.4624066Z   test_cuda_future_can_extract_list_with_cuda_sparse_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29771
2022-05-18T03:44:28.4646442Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29772
2022-05-18T03:44:28.4669665Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29773
2022-05-18T03:44:28.4694614Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 29774
2022-05-18T03:44:29.1209006Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzsmu5ur5
2022-05-18T03:44:29.1209776Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzsmu5ur5/_remote_module_non_scriptable.py
2022-05-18T03:44:29.1437598Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp29tiyr53
2022-05-18T03:44:29.1438379Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp29tiyr53/_remote_module_non_scriptable.py
2022-05-18T03:44:29.1677820Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt19s209z
2022-05-18T03:44:29.1678730Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt19s209z/_remote_module_non_scriptable.py
2022-05-18T03:44:29.1804018Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi2ks4ed8
2022-05-18T03:44:29.1805509Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi2ks4ed8/_remote_module_non_scriptable.py
2022-05-18T03:44:29.3705168Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:44:29.3926055Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:44:29.4148172Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:44:29.4292177Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:44:29.5728713Z skip: Need at least 1 CUDA device (1.429s)
2022-05-18T03:44:29.5729024Z 
2022-05-18T03:44:29.5729376Z ----------------------------------------------------------------------
2022-05-18T03:44:29.5729644Z Ran 1 test in 1.429s
2022-05-18T03:44:29.5729746Z 
2022-05-18T03:44:29.5729838Z OK (skipped=1)
2022-05-18T03:44:29.5729949Z 
2022-05-18T03:44:29.5730037Z Generating XML reports...
2022-05-18T03:44:29.5760336Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518034428.xml
2022-05-18T03:44:30.3187985Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwsmlf3lo
2022-05-18T03:44:30.3188648Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwsmlf3lo/_remote_module_non_scriptable.py
2022-05-18T03:44:30.5662604Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T03:44:30.5672648Z 
2022-05-18T03:44:30.5673054Z Running tests...
2022-05-18T03:44:30.5673483Z ----------------------------------------------------------------------
2022-05-18T03:44:30.8815840Z   test_cuda_future_can_extract_list_with_cuda_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29826
2022-05-18T03:44:30.8837065Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29827
2022-05-18T03:44:30.8859813Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29828
2022-05-18T03:44:30.8884385Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 29829
2022-05-18T03:44:31.5253413Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptjy824rm
2022-05-18T03:44:31.5254140Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptjy824rm/_remote_module_non_scriptable.py
2022-05-18T03:44:31.5357849Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_ei11sss
2022-05-18T03:44:31.5360370Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_ei11sss/_remote_module_non_scriptable.py
2022-05-18T03:44:31.5509877Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp24z943w6
2022-05-18T03:44:31.5512701Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp24z943w6/_remote_module_non_scriptable.py
2022-05-18T03:44:31.5558835Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8npxnwia
2022-05-18T03:44:31.5560789Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8npxnwia/_remote_module_non_scriptable.py
2022-05-18T03:44:31.7742454Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:44:31.7835061Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:44:31.7974462Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:44:31.8033600Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:44:31.9919068Z skip: Need at least 1 CUDA device (1.424s)
2022-05-18T03:44:31.9919377Z 
2022-05-18T03:44:31.9919892Z ----------------------------------------------------------------------
2022-05-18T03:44:31.9920195Z Ran 1 test in 1.425s
2022-05-18T03:44:31.9920313Z 
2022-05-18T03:44:31.9920386Z OK (skipped=1)
2022-05-18T03:44:31.9920493Z 
2022-05-18T03:44:31.9920578Z Generating XML reports...
2022-05-18T03:44:31.9952666Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518034430.xml
2022-05-18T03:44:32.7365748Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnfdquqj5
2022-05-18T03:44:32.7366484Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnfdquqj5/_remote_module_non_scriptable.py
2022-05-18T03:44:32.9831920Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T03:44:32.9840861Z 
2022-05-18T03:44:32.9840994Z Running tests...
2022-05-18T03:44:32.9841584Z ----------------------------------------------------------------------
2022-05-18T03:44:33.2939766Z   test_cuda_future_device_as_device (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29881
2022-05-18T03:44:33.2962337Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29882
2022-05-18T03:44:33.2985167Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29883
2022-05-18T03:44:33.3009077Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 29884
2022-05-18T03:44:33.9382434Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgua5ck93
2022-05-18T03:44:33.9383550Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgua5ck93/_remote_module_non_scriptable.py
2022-05-18T03:44:33.9429407Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp89myzojs
2022-05-18T03:44:33.9430597Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp89myzojs/_remote_module_non_scriptable.py
2022-05-18T03:44:34.0229240Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkt3jvb_9
2022-05-18T03:44:34.0230198Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkt3jvb_9/_remote_module_non_scriptable.py
2022-05-18T03:44:34.0238605Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnkb6aeeo
2022-05-18T03:44:34.0240030Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnkb6aeeo/_remote_module_non_scriptable.py
2022-05-18T03:44:34.1876224Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:44:34.1900321Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:44:34.2708651Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:44:34.2761454Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:44:34.5044651Z skip: Need at least 1 CUDA device (1.520s)
2022-05-18T03:44:34.5045048Z 
2022-05-18T03:44:34.5045407Z ----------------------------------------------------------------------
2022-05-18T03:44:34.5045838Z Ran 1 test in 1.520s
2022-05-18T03:44:34.5046036Z 
2022-05-18T03:44:34.5046292Z OK (skipped=1)
2022-05-18T03:44:34.5046452Z 
2022-05-18T03:44:34.5046544Z Generating XML reports...
2022-05-18T03:44:34.5078496Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518034432.xml
2022-05-18T03:44:35.2451849Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_329rxpr
2022-05-18T03:44:35.2452553Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_329rxpr/_remote_module_non_scriptable.py
2022-05-18T03:44:35.4926511Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T03:44:35.4936031Z 
2022-05-18T03:44:35.4936159Z Running tests...
2022-05-18T03:44:35.4936742Z ----------------------------------------------------------------------
2022-05-18T03:44:35.8046450Z   test_cuda_future_device_as_int (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29936
2022-05-18T03:44:35.8068524Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29937
2022-05-18T03:44:35.8091065Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29938
2022-05-18T03:44:35.8116237Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 29939
2022-05-18T03:44:36.4363927Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfrl4sxyx
2022-05-18T03:44:36.4365209Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfrl4sxyx/_remote_module_non_scriptable.py
2022-05-18T03:44:36.4505975Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp_w8oyr5
2022-05-18T03:44:36.4507037Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp_w8oyr5/_remote_module_non_scriptable.py
2022-05-18T03:44:36.4528033Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6m1v3qzh
2022-05-18T03:44:36.4529212Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6m1v3qzh/_remote_module_non_scriptable.py
2022-05-18T03:44:36.4633058Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3ytl12j7
2022-05-18T03:44:36.4634474Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3ytl12j7/_remote_module_non_scriptable.py
2022-05-18T03:44:36.6823926Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:44:36.6972389Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:44:36.7022554Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:44:36.7119469Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:44:36.9149468Z skip: Need at least 1 CUDA device (1.421s)
2022-05-18T03:44:36.9149738Z 
2022-05-18T03:44:36.9150264Z ----------------------------------------------------------------------
2022-05-18T03:44:36.9150691Z Ran 1 test in 1.421s
2022-05-18T03:44:36.9150861Z 
2022-05-18T03:44:36.9150973Z OK (skipped=1)
2022-05-18T03:44:36.9151085Z 
2022-05-18T03:44:36.9151212Z Generating XML reports...
2022-05-18T03:44:36.9182833Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518034435.xml
2022-05-18T03:44:37.6550658Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyfy3ft_j
2022-05-18T03:44:37.6551458Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyfy3ft_j/_remote_module_non_scriptable.py
2022-05-18T03:44:37.9010422Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T03:44:37.9020610Z 
2022-05-18T03:44:37.9021104Z Running tests...
2022-05-18T03:44:37.9021743Z ----------------------------------------------------------------------
2022-05-18T03:44:38.2118849Z   test_cuda_future_device_as_str (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29991
2022-05-18T03:44:38.2141080Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29992
2022-05-18T03:44:38.2164361Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29993
2022-05-18T03:44:38.2188989Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 29994
2022-05-18T03:44:38.8006968Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0vkct862
2022-05-18T03:44:38.8008115Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0vkct862/_remote_module_non_scriptable.py
2022-05-18T03:44:38.8028901Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2qq6rxed
2022-05-18T03:44:38.8030640Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2qq6rxed/_remote_module_non_scriptable.py
2022-05-18T03:44:38.8428507Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy8gykj42
2022-05-18T03:44:38.8429268Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy8gykj42/_remote_module_non_scriptable.py
2022-05-18T03:44:38.8512087Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp39s1xtkh
2022-05-18T03:44:38.8513473Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp39s1xtkh/_remote_module_non_scriptable.py
2022-05-18T03:44:39.0498915Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:44:39.0513701Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:44:39.0901111Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:44:39.0996673Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:44:39.3222974Z skip: Need at least 1 CUDA device (1.420s)
2022-05-18T03:44:39.3223328Z 
2022-05-18T03:44:39.3223924Z ----------------------------------------------------------------------
2022-05-18T03:44:39.3224304Z Ran 1 test in 1.420s
2022-05-18T03:44:39.3224420Z 
2022-05-18T03:44:39.3224493Z OK (skipped=1)
2022-05-18T03:44:39.3224588Z 
2022-05-18T03:44:39.3224681Z Generating XML reports...
2022-05-18T03:44:39.3255541Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518034437.xml
2022-05-18T03:44:40.0684878Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv1uz1i_7
2022-05-18T03:44:40.0685624Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv1uz1i_7/_remote_module_non_scriptable.py
2022-05-18T03:44:40.3154758Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T03:44:40.3164788Z 
2022-05-18T03:44:40.3165124Z Running tests...
2022-05-18T03:44:40.3165784Z ----------------------------------------------------------------------
2022-05-18T03:44:40.6308476Z   test_cuda_future_device_not_cuda (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30046
2022-05-18T03:44:40.6330288Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30047
2022-05-18T03:44:40.6353430Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30048
2022-05-18T03:44:40.6377855Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 30049
2022-05-18T03:44:41.2310667Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0wcwltp1
2022-05-18T03:44:41.2311650Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0wcwltp1/_remote_module_non_scriptable.py
2022-05-18T03:44:41.2504727Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn9al0lyp
2022-05-18T03:44:41.2507096Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn9al0lyp/_remote_module_non_scriptable.py
2022-05-18T03:44:41.2558586Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi2jifff0
2022-05-18T03:44:41.2559862Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi2jifff0/_remote_module_non_scriptable.py
2022-05-18T03:44:41.2613482Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8i1hy94z
2022-05-18T03:44:41.2615495Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8i1hy94z/_remote_module_non_scriptable.py
2022-05-18T03:44:41.4813021Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:44:41.4991825Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:44:41.5019510Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:44:41.5112057Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:44:41.7412200Z skip: Need at least 1 CUDA device (1.424s)
2022-05-18T03:44:41.7412580Z 
2022-05-18T03:44:41.7413039Z ----------------------------------------------------------------------
2022-05-18T03:44:41.7413296Z Ran 1 test in 1.425s
2022-05-18T03:44:41.7413413Z 
2022-05-18T03:44:41.7413486Z OK (skipped=1)
2022-05-18T03:44:41.7413581Z 
2022-05-18T03:44:41.7413671Z Generating XML reports...
2022-05-18T03:44:41.7447484Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518034440.xml
2022-05-18T03:44:42.4887465Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5jtm_gks
2022-05-18T03:44:42.4888394Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5jtm_gks/_remote_module_non_scriptable.py
2022-05-18T03:44:42.7364966Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T03:44:42.7374742Z 
2022-05-18T03:44:42.7375018Z Running tests...
2022-05-18T03:44:42.7375722Z ----------------------------------------------------------------------
2022-05-18T03:44:43.0482398Z   test_cuda_future_modify_tensor_inplace (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30101
2022-05-18T03:44:43.0504656Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30102
2022-05-18T03:44:43.0527469Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30103
2022-05-18T03:44:43.0551127Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 30104
2022-05-18T03:44:43.7512824Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqcnvo3we
2022-05-18T03:44:43.7513616Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqcnvo3we/_remote_module_non_scriptable.py
2022-05-18T03:44:43.7516185Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsxov9ipz
2022-05-18T03:44:43.7518522Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsxov9ipz/_remote_module_non_scriptable.py
2022-05-18T03:44:43.8013985Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsmzp_q_4
2022-05-18T03:44:43.8014792Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsmzp_q_4/_remote_module_non_scriptable.py
2022-05-18T03:44:43.8188119Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp74l88d57
2022-05-18T03:44:43.8188840Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp74l88d57/_remote_module_non_scriptable.py
2022-05-18T03:44:43.9977018Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:44:44.0013492Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:44:44.0514821Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:44:44.0767566Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:44:44.2587963Z skip: Need at least 1 CUDA device (1.521s)
2022-05-18T03:44:44.2588290Z 
2022-05-18T03:44:44.2588830Z ----------------------------------------------------------------------
2022-05-18T03:44:44.2589106Z Ran 1 test in 1.521s
2022-05-18T03:44:44.2589219Z 
2022-05-18T03:44:44.2589292Z OK (skipped=1)
2022-05-18T03:44:44.2589400Z 
2022-05-18T03:44:44.2589485Z Generating XML reports...
2022-05-18T03:44:44.2622304Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518034442.xml
2022-05-18T03:44:45.0022468Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph2y1nwfs
2022-05-18T03:44:45.0023562Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph2y1nwfs/_remote_module_non_scriptable.py
2022-05-18T03:44:45.2477730Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T03:44:45.2487732Z 
2022-05-18T03:44:45.2488033Z Running tests...
2022-05-18T03:44:45.2488665Z ----------------------------------------------------------------------
2022-05-18T03:44:45.5605998Z   test_cuda_future_replace_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30156
2022-05-18T03:44:45.5628650Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30157
2022-05-18T03:44:45.5651383Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30158
2022-05-18T03:44:45.5675059Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 30159
2022-05-18T03:44:46.1885853Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyw_55y34
2022-05-18T03:44:46.1886881Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyw_55y34/_remote_module_non_scriptable.py
2022-05-18T03:44:46.2184097Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9j0ttt8j
2022-05-18T03:44:46.2185076Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9j0ttt8j/_remote_module_non_scriptable.py
2022-05-18T03:44:46.2277718Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkl8ymzaz
2022-05-18T03:44:46.2279009Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkl8ymzaz/_remote_module_non_scriptable.py
2022-05-18T03:44:46.2289252Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3nuaf3a_
2022-05-18T03:44:46.2291672Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3nuaf3a_/_remote_module_non_scriptable.py
2022-05-18T03:44:46.4403314Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:44:46.4759583Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:44:46.4773270Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:44:46.4774792Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:44:46.6709675Z skip: Need at least 1 CUDA device (1.422s)
2022-05-18T03:44:46.6709967Z 
2022-05-18T03:44:46.6710355Z ----------------------------------------------------------------------
2022-05-18T03:44:46.6710613Z Ran 1 test in 1.422s
2022-05-18T03:44:46.6710718Z 
2022-05-18T03:44:46.6710792Z OK (skipped=1)
2022-05-18T03:44:46.6710915Z 
2022-05-18T03:44:46.6711048Z Generating XML reports...
2022-05-18T03:44:46.6742796Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518034445.xml
2022-05-18T03:44:47.4212082Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprd_skwaw
2022-05-18T03:44:47.4213056Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprd_skwaw/_remote_module_non_scriptable.py
2022-05-18T03:44:47.6691200Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T03:44:47.6700997Z 
2022-05-18T03:44:47.6701632Z Running tests...
2022-05-18T03:44:47.6702022Z ----------------------------------------------------------------------
2022-05-18T03:44:47.9820649Z   test_cuda_future_value_on_bad_device (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30211
2022-05-18T03:44:47.9842999Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30212
2022-05-18T03:44:47.9865958Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30213
2022-05-18T03:44:47.9889910Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 30214
2022-05-18T03:44:48.6367074Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbesnwaps
2022-05-18T03:44:48.6367905Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbesnwaps/_remote_module_non_scriptable.py
2022-05-18T03:44:48.6746421Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnl7j4g8l
2022-05-18T03:44:48.6747516Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnl7j4g8l/_remote_module_non_scriptable.py
2022-05-18T03:44:48.6765152Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyrr3e5d6
2022-05-18T03:44:48.6767059Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyrr3e5d6/_remote_module_non_scriptable.py
2022-05-18T03:44:48.6950605Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw3nsmn17
2022-05-18T03:44:48.6951442Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw3nsmn17/_remote_module_non_scriptable.py
2022-05-18T03:44:48.8864219Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:44:48.9221307Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:44:48.9224807Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:44:48.9439525Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:44:49.0925650Z skip: Need at least 2 CUDA devices (1.422s)
2022-05-18T03:44:49.0926197Z 
2022-05-18T03:44:49.0926752Z ----------------------------------------------------------------------
2022-05-18T03:44:49.0927243Z Ran 1 test in 1.422s
2022-05-18T03:44:49.0927456Z 
2022-05-18T03:44:49.0927570Z OK (skipped=1)
2022-05-18T03:44:49.0927684Z 
2022-05-18T03:44:49.0927771Z Generating XML reports...
2022-05-18T03:44:49.0958806Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518034447.xml
2022-05-18T03:44:49.8396716Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo53ue0e9
2022-05-18T03:44:49.8397521Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo53ue0e9/_remote_module_non_scriptable.py
2022-05-18T03:44:50.0867165Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T03:44:50.0876885Z 
2022-05-18T03:44:50.0877323Z Running tests...
2022-05-18T03:44:50.0877734Z ----------------------------------------------------------------------
2022-05-18T03:44:50.3963680Z   test_custom_stream (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30266
2022-05-18T03:44:50.3986531Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30267
2022-05-18T03:44:50.4009039Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30268
2022-05-18T03:44:50.4033605Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 30269
2022-05-18T03:44:51.1166660Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5xv4phsh
2022-05-18T03:44:51.1167983Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5xv4phsh/_remote_module_non_scriptable.py
2022-05-18T03:44:51.1345514Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzcobogj_
2022-05-18T03:44:51.1346612Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzcobogj_/_remote_module_non_scriptable.py
2022-05-18T03:44:51.1436615Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptslpye0c
2022-05-18T03:44:51.1437721Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptslpye0c/_remote_module_non_scriptable.py
2022-05-18T03:44:51.1469667Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcepyc4fd
2022-05-18T03:44:51.1472203Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcepyc4fd/_remote_module_non_scriptable.py
2022-05-18T03:44:51.3629020Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:44:51.3840392Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:44:51.3919030Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:44:51.3962696Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:44:51.6070922Z skip: Need at least 2 CUDA devices (1.519s)
2022-05-18T03:44:51.6071216Z 
2022-05-18T03:44:51.6071725Z ----------------------------------------------------------------------
2022-05-18T03:44:51.6072183Z Ran 1 test in 1.519s
2022-05-18T03:44:51.6072392Z 
2022-05-18T03:44:51.6072511Z OK (skipped=1)
2022-05-18T03:44:51.6072628Z 
2022-05-18T03:44:51.6072714Z Generating XML reports...
2022-05-18T03:44:51.6103590Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518034450.xml
2022-05-18T03:44:52.3501809Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdf6_qq4b
2022-05-18T03:44:52.3502543Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdf6_qq4b/_remote_module_non_scriptable.py
2022-05-18T03:44:52.5990873Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T03:44:52.6000545Z 
2022-05-18T03:44:52.6000640Z Running tests...
2022-05-18T03:44:52.6001215Z ----------------------------------------------------------------------
2022-05-18T03:44:52.9108833Z   test_custom_stream_multi (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30321
2022-05-18T03:44:52.9130951Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30322
2022-05-18T03:44:52.9153723Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30323
2022-05-18T03:44:52.9176878Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 30324
2022-05-18T03:44:53.5458146Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpomc8nfpf
2022-05-18T03:44:53.5508416Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpomc8nfpf/_remote_module_non_scriptable.py
2022-05-18T03:44:53.5550776Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdw3438ef
2022-05-18T03:44:53.5551818Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbgo90yyt
2022-05-18T03:44:53.5552438Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdw3438ef/_remote_module_non_scriptable.py
2022-05-18T03:44:53.5553605Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbgo90yyt/_remote_module_non_scriptable.py
2022-05-18T03:44:53.5593271Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpacuehf25
2022-05-18T03:44:53.5594959Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpacuehf25/_remote_module_non_scriptable.py
2022-05-18T03:44:53.7934210Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:44:53.8015104Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:44:53.8024814Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:44:53.8138795Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:44:54.0212901Z skip: Need at least 2 CUDA devices (1.421s)
2022-05-18T03:44:54.0213218Z 
2022-05-18T03:44:54.0213715Z ----------------------------------------------------------------------
2022-05-18T03:44:54.0213966Z Ran 1 test in 1.421s
2022-05-18T03:44:54.0214080Z 
2022-05-18T03:44:54.0214155Z OK (skipped=1)
2022-05-18T03:44:54.0214266Z 
2022-05-18T03:44:54.0214363Z Generating XML reports...
2022-05-18T03:44:54.0246043Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518034452.xml
2022-05-18T03:44:54.7617971Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp50n539oj
2022-05-18T03:44:54.7618903Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp50n539oj/_remote_module_non_scriptable.py
2022-05-18T03:44:55.0092732Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T03:44:55.0102838Z 
2022-05-18T03:44:55.0103650Z Running tests...
2022-05-18T03:44:55.0104230Z ----------------------------------------------------------------------
2022-05-18T03:44:55.3301968Z   test_custom_stream_nested (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30376
2022-05-18T03:44:55.3325188Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30377
2022-05-18T03:44:55.3348508Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30378
2022-05-18T03:44:55.3372745Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 30379
2022-05-18T03:44:56.0108892Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvuq1qx_d
2022-05-18T03:44:56.0111833Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvuq1qx_d/_remote_module_non_scriptable.py
2022-05-18T03:44:56.0146732Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcjwoi21b
2022-05-18T03:44:56.0148672Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcjwoi21b/_remote_module_non_scriptable.py
2022-05-18T03:44:56.0561995Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptt8e0qu5
2022-05-18T03:44:56.0562810Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptt8e0qu5/_remote_module_non_scriptable.py
2022-05-18T03:44:56.0689966Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkjsd0w1f
2022-05-18T03:44:56.0691491Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkjsd0w1f/_remote_module_non_scriptable.py
2022-05-18T03:44:56.2570385Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:44:56.2643313Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:44:56.3048932Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:44:56.3186913Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:44:56.5408406Z skip: Need at least 2 CUDA devices (1.530s)
2022-05-18T03:44:56.5408701Z 
2022-05-18T03:44:56.5409210Z ----------------------------------------------------------------------
2022-05-18T03:44:56.5409611Z Ran 1 test in 1.531s
2022-05-18T03:44:56.5409727Z 
2022-05-18T03:44:56.5410001Z OK (skipped=1)
2022-05-18T03:44:56.5410113Z 
2022-05-18T03:44:56.5410202Z Generating XML reports...
2022-05-18T03:44:56.5443758Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518034455.xml
2022-05-18T03:44:57.2861801Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk5kwde5l
2022-05-18T03:44:57.2862530Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk5kwde5l/_remote_module_non_scriptable.py
2022-05-18T03:44:57.5359460Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T03:44:57.5369777Z 
2022-05-18T03:44:57.5370138Z Running tests...
2022-05-18T03:44:57.5370770Z ----------------------------------------------------------------------
2022-05-18T03:44:57.8466884Z   test_custom_stream_nested_multi (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30431
2022-05-18T03:44:57.8488888Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30432
2022-05-18T03:44:57.8512371Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30433
2022-05-18T03:44:57.8535995Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 30434
2022-05-18T03:44:58.4897808Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp382vv2t4
2022-05-18T03:44:58.4898881Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp382vv2t4/_remote_module_non_scriptable.py
2022-05-18T03:44:58.4900765Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9prhar5r
2022-05-18T03:44:58.4904166Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9prhar5r/_remote_module_non_scriptable.py
2022-05-18T03:44:58.5865333Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmerr6u24
2022-05-18T03:44:58.5866131Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmerr6u24/_remote_module_non_scriptable.py
2022-05-18T03:44:58.6395703Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdglzum9s
2022-05-18T03:44:58.6397395Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdglzum9s/_remote_module_non_scriptable.py
2022-05-18T03:44:58.7360833Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:44:58.7401298Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:44:58.8854183Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:44:58.9182530Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:44:59.0571946Z skip: Need at least 2 CUDA devices (1.520s)
2022-05-18T03:44:59.0572556Z 
2022-05-18T03:44:59.0573314Z ----------------------------------------------------------------------
2022-05-18T03:44:59.0573858Z Ran 1 test in 1.520s
2022-05-18T03:44:59.0573988Z 
2022-05-18T03:44:59.0574074Z OK (skipped=1)
2022-05-18T03:44:59.0574183Z 
2022-05-18T03:44:59.0574270Z Generating XML reports...
2022-05-18T03:44:59.0605434Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518034457.xml
2022-05-18T03:44:59.7988433Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8peoembx
2022-05-18T03:44:59.7989205Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8peoembx/_remote_module_non_scriptable.py
2022-05-18T03:45:00.0453053Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T03:45:00.0462634Z 
2022-05-18T03:45:00.0462867Z Running tests...
2022-05-18T03:45:00.0463496Z ----------------------------------------------------------------------
2022-05-18T03:45:00.3576094Z   test_device_map_cpu (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30486
2022-05-18T03:45:00.3599476Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30487
2022-05-18T03:45:00.3622355Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30488
2022-05-18T03:45:00.3646051Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 30489
2022-05-18T03:45:01.0488277Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc04b2hjc
2022-05-18T03:45:01.0488977Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc04b2hjc/_remote_module_non_scriptable.py
2022-05-18T03:45:01.0528049Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8gwm9t5d
2022-05-18T03:45:01.0528983Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8gwm9t5d/_remote_module_non_scriptable.py
2022-05-18T03:45:01.0988561Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2q3kusz4
2022-05-18T03:45:01.0989575Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2q3kusz4/_remote_module_non_scriptable.py
2022-05-18T03:45:01.1104004Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpejxr3w84
2022-05-18T03:45:01.1105569Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpejxr3w84/_remote_module_non_scriptable.py
2022-05-18T03:45:01.2981342Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:45:01.3034905Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:45:01.3469632Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:45:01.3601405Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:45:01.8687347Z ok (1.822s)
2022-05-18T03:45:01.8687611Z 
2022-05-18T03:45:01.8688148Z ----------------------------------------------------------------------
2022-05-18T03:45:01.8688466Z Ran 1 test in 1.822s
2022-05-18T03:45:01.8688583Z 
2022-05-18T03:45:01.8688645Z OK
2022-05-18T03:45:01.8688739Z 
2022-05-18T03:45:01.8688842Z Generating XML reports...
2022-05-18T03:45:01.8721135Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518034500.xml
2022-05-18T03:45:02.6578631Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpam6ik6m3
2022-05-18T03:45:02.6579691Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpam6ik6m3/_remote_module_non_scriptable.py
2022-05-18T03:45:02.9063927Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T03:45:02.9073582Z 
2022-05-18T03:45:02.9074038Z Running tests...
2022-05-18T03:45:02.9074428Z ----------------------------------------------------------------------
2022-05-18T03:45:03.2218579Z   test_device_map_cpu_to_gpu_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30705
2022-05-18T03:45:03.2240775Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30706
2022-05-18T03:45:03.2264044Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30707
2022-05-18T03:45:03.2287762Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 30708
2022-05-18T03:45:03.8301627Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc3g9mlc7
2022-05-18T03:45:03.8302417Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc3g9mlc7/_remote_module_non_scriptable.py
2022-05-18T03:45:03.8512956Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4mcjifrf
2022-05-18T03:45:03.8513774Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4mcjifrf/_remote_module_non_scriptable.py
2022-05-18T03:45:03.8619310Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1exu2sqi
2022-05-18T03:45:03.8621431Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1exu2sqi/_remote_module_non_scriptable.py
2022-05-18T03:45:03.8928469Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi0yma2f4
2022-05-18T03:45:03.8929598Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi0yma2f4/_remote_module_non_scriptable.py
2022-05-18T03:45:04.0774803Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:45:04.0991963Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:45:04.1255084Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:45:04.1415500Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:45:04.3323001Z skip: Need at least 1 CUDA device (1.425s)
2022-05-18T03:45:04.3323311Z 
2022-05-18T03:45:04.3323852Z ----------------------------------------------------------------------
2022-05-18T03:45:04.3324100Z Ran 1 test in 1.425s
2022-05-18T03:45:04.3324218Z 
2022-05-18T03:45:04.3324298Z OK (skipped=1)
2022-05-18T03:45:04.3324407Z 
2022-05-18T03:45:04.3324493Z Generating XML reports...
2022-05-18T03:45:04.3356194Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518034502.xml
2022-05-18T03:45:05.0776627Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_7q21lm9
2022-05-18T03:45:05.0777365Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_7q21lm9/_remote_module_non_scriptable.py
2022-05-18T03:45:05.3251138Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T03:45:05.3261069Z 
2022-05-18T03:45:05.3261436Z Running tests...
2022-05-18T03:45:05.3261865Z ----------------------------------------------------------------------
2022-05-18T03:45:05.6367508Z   test_device_map_cpu_to_gpu_non_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30760
2022-05-18T03:45:05.6389974Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30761
2022-05-18T03:45:05.6412358Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30762
2022-05-18T03:45:05.6437010Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 30763
2022-05-18T03:45:06.2804906Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpycke7u74
2022-05-18T03:45:06.2807125Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpycke7u74/_remote_module_non_scriptable.py
2022-05-18T03:45:06.2931024Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkam02c0e
2022-05-18T03:45:06.2932223Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkam02c0e/_remote_module_non_scriptable.py
2022-05-18T03:45:06.3266988Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo1zwiqiq
2022-05-18T03:45:06.3268121Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo1zwiqiq/_remote_module_non_scriptable.py
2022-05-18T03:45:06.3448981Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprupxv8mn
2022-05-18T03:45:06.3449914Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprupxv8mn/_remote_module_non_scriptable.py
2022-05-18T03:45:06.5287828Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:45:06.5456896Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:45:06.5826567Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:45:06.5951232Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:45:06.7471942Z skip: Need at least 2 CUDA devices (1.421s)
2022-05-18T03:45:06.7472237Z 
2022-05-18T03:45:06.7472971Z ----------------------------------------------------------------------
2022-05-18T03:45:06.7473233Z Ran 1 test in 1.421s
2022-05-18T03:45:06.7473347Z 
2022-05-18T03:45:06.7473421Z OK (skipped=1)
2022-05-18T03:45:06.7473535Z 
2022-05-18T03:45:06.7473607Z Generating XML reports...
2022-05-18T03:45:06.7504837Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518034505.xml
2022-05-18T03:45:07.4957763Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmr3n0w64
2022-05-18T03:45:07.4958561Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmr3n0w64/_remote_module_non_scriptable.py
2022-05-18T03:45:07.7432576Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T03:45:07.7442447Z 
2022-05-18T03:45:07.7442924Z Running tests...
2022-05-18T03:45:07.7443367Z ----------------------------------------------------------------------
2022-05-18T03:45:08.0563537Z   test_device_map_gpu_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30815
2022-05-18T03:45:08.0585685Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30816
2022-05-18T03:45:08.0608777Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30817
2022-05-18T03:45:08.0632011Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 30818
2022-05-18T03:45:08.6653320Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq3emxzxb
2022-05-18T03:45:08.6654662Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq3emxzxb/_remote_module_non_scriptable.py
2022-05-18T03:45:08.6764593Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpptfkhi5d
2022-05-18T03:45:08.6766579Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpptfkhi5d/_remote_module_non_scriptable.py
2022-05-18T03:45:08.6768826Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1ey2tjeu
2022-05-18T03:45:08.6771933Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1ey2tjeu/_remote_module_non_scriptable.py
2022-05-18T03:45:08.6930162Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplvglo0ov
2022-05-18T03:45:08.6931356Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplvglo0ov/_remote_module_non_scriptable.py
2022-05-18T03:45:08.9140446Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:45:08.9235188Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:45:08.9270788Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:45:08.9423402Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:45:09.1667299Z skip: Need at least 2 CUDA devices (1.422s)
2022-05-18T03:45:09.1667480Z 
2022-05-18T03:45:09.1667896Z ----------------------------------------------------------------------
2022-05-18T03:45:09.1668135Z Ran 1 test in 1.422s
2022-05-18T03:45:09.1668278Z 
2022-05-18T03:45:09.1668395Z OK (skipped=1)
2022-05-18T03:45:09.1668509Z 
2022-05-18T03:45:09.1668595Z Generating XML reports...
2022-05-18T03:45:09.1700317Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518034507.xml
2022-05-18T03:45:09.9108436Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6zfcyhj9
2022-05-18T03:45:09.9109511Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6zfcyhj9/_remote_module_non_scriptable.py
2022-05-18T03:45:10.1581249Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T03:45:10.1591377Z 
2022-05-18T03:45:10.1591512Z Running tests...
2022-05-18T03:45:10.1592268Z ----------------------------------------------------------------------
2022-05-18T03:45:10.4720566Z   test_device_map_gpu_default_to_non_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30870
2022-05-18T03:45:10.4742648Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30871
2022-05-18T03:45:10.4765891Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30872
2022-05-18T03:45:10.4789419Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 30873
2022-05-18T03:45:11.1416269Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6f87e45h
2022-05-18T03:45:11.1417046Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6f87e45h/_remote_module_non_scriptable.py
2022-05-18T03:45:11.1576513Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwgmzy6no
2022-05-18T03:45:11.1577541Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwgmzy6no/_remote_module_non_scriptable.py
2022-05-18T03:45:11.1669572Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeb5bhsip
2022-05-18T03:45:11.1670837Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeb5bhsip/_remote_module_non_scriptable.py
2022-05-18T03:45:11.1785556Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv96zqzmz
2022-05-18T03:45:11.1786887Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv96zqzmz/_remote_module_non_scriptable.py
2022-05-18T03:45:11.3908370Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:45:11.4057252Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:45:11.4149428Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:45:11.4271794Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:45:11.5826132Z skip: Need at least 2 CUDA devices (1.423s)
2022-05-18T03:45:11.5826382Z 
2022-05-18T03:45:11.5826687Z ----------------------------------------------------------------------
2022-05-18T03:45:11.5826970Z Ran 1 test in 1.423s
2022-05-18T03:45:11.5827124Z 
2022-05-18T03:45:11.5827199Z OK (skipped=1)
2022-05-18T03:45:11.5827295Z 
2022-05-18T03:45:11.5827385Z Generating XML reports...
2022-05-18T03:45:11.5857934Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518034510.xml
2022-05-18T03:45:12.4054034Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk0dses_x
2022-05-18T03:45:12.4054874Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk0dses_x/_remote_module_non_scriptable.py
2022-05-18T03:45:12.6518096Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T03:45:12.6527558Z 
2022-05-18T03:45:12.6528010Z Running tests...
2022-05-18T03:45:12.6528599Z ----------------------------------------------------------------------
2022-05-18T03:45:12.9686402Z   test_device_map_gpu_mixed_1 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30925
2022-05-18T03:45:12.9707931Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30926
2022-05-18T03:45:12.9730145Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30927
2022-05-18T03:45:12.9754030Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 30928
2022-05-18T03:45:13.5406776Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyi1b1764
2022-05-18T03:45:13.5407796Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyi1b1764/_remote_module_non_scriptable.py
2022-05-18T03:45:13.6081187Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx5wo096o
2022-05-18T03:45:13.6081916Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9loqcbmo
2022-05-18T03:45:13.6082644Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx5wo096o/_remote_module_non_scriptable.py
2022-05-18T03:45:13.6083320Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9loqcbmo/_remote_module_non_scriptable.py
2022-05-18T03:45:13.6268156Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1crsw076
2022-05-18T03:45:13.6269363Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1crsw076/_remote_module_non_scriptable.py
2022-05-18T03:45:13.7907007Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:45:13.8553391Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:45:13.8579007Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:45:13.8822117Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:45:14.0789016Z skip: Need at least 2 CUDA devices (1.426s)
2022-05-18T03:45:14.0789280Z 
2022-05-18T03:45:14.0789588Z ----------------------------------------------------------------------
2022-05-18T03:45:14.0789844Z Ran 1 test in 1.426s
2022-05-18T03:45:14.0789999Z 
2022-05-18T03:45:14.0790082Z OK (skipped=1)
2022-05-18T03:45:14.0790193Z 
2022-05-18T03:45:14.0790278Z Generating XML reports...
2022-05-18T03:45:14.0821455Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518034512.xml
2022-05-18T03:45:14.8255258Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy7due6_v
2022-05-18T03:45:14.8255952Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy7due6_v/_remote_module_non_scriptable.py
2022-05-18T03:45:15.0735335Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T03:45:15.0745299Z 
2022-05-18T03:45:15.0745572Z Running tests...
2022-05-18T03:45:15.0746024Z ----------------------------------------------------------------------
2022-05-18T03:45:15.3867590Z   test_device_map_gpu_mixed_2 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30980
2022-05-18T03:45:15.3890467Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30981
2022-05-18T03:45:15.3913354Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30982
2022-05-18T03:45:15.3936152Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 30983
2022-05-18T03:45:16.0351172Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp739n_5qr
2022-05-18T03:45:16.0351938Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp739n_5qr/_remote_module_non_scriptable.py
2022-05-18T03:45:16.0436789Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi_g0_eiv
2022-05-18T03:45:16.0437962Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi_g0_eiv/_remote_module_non_scriptable.py
2022-05-18T03:45:16.0442222Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp26pusaei
2022-05-18T03:45:16.0443951Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp26pusaei/_remote_module_non_scriptable.py
2022-05-18T03:45:16.0553504Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpupfkwlvo
2022-05-18T03:45:16.0555386Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpupfkwlvo/_remote_module_non_scriptable.py
2022-05-18T03:45:16.2832059Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:45:16.2903634Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:45:16.2920644Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:45:16.3062588Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:45:16.4971274Z skip: Need at least 2 CUDA devices (1.422s)
2022-05-18T03:45:16.4971595Z 
2022-05-18T03:45:16.4972098Z ----------------------------------------------------------------------
2022-05-18T03:45:16.4972339Z Ran 1 test in 1.423s
2022-05-18T03:45:16.4972455Z 
2022-05-18T03:45:16.4972530Z OK (skipped=1)
2022-05-18T03:45:16.4972638Z 
2022-05-18T03:45:16.4972725Z Generating XML reports...
2022-05-18T03:45:16.5004216Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518034515.xml
2022-05-18T03:45:17.2301792Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfhpw9yql
2022-05-18T03:45:17.2302534Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfhpw9yql/_remote_module_non_scriptable.py
2022-05-18T03:45:17.4780805Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T03:45:17.4791402Z 
2022-05-18T03:45:17.4791537Z Running tests...
2022-05-18T03:45:17.4791982Z ----------------------------------------------------------------------
2022-05-18T03:45:17.7905040Z   test_device_map_gpu_mixed_3 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31035
2022-05-18T03:45:17.7926852Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31036
2022-05-18T03:45:17.7949343Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31037
2022-05-18T03:45:17.7972541Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 31038
2022-05-18T03:45:18.4604581Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjlp3gwaz
2022-05-18T03:45:18.4605645Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjlp3gwaz/_remote_module_non_scriptable.py
2022-05-18T03:45:18.4683923Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq82a2jb4
2022-05-18T03:45:18.4686406Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq82a2jb4/_remote_module_non_scriptable.py
2022-05-18T03:45:18.5010210Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyzgr8urc
2022-05-18T03:45:18.5012718Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyzgr8urc/_remote_module_non_scriptable.py
2022-05-18T03:45:18.5121525Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpatf0a2nd
2022-05-18T03:45:18.5122621Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpatf0a2nd/_remote_module_non_scriptable.py
2022-05-18T03:45:18.7124528Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:45:18.7169804Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:45:18.7518842Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:45:18.7685028Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:45:18.9007862Z skip: Need at least 2 CUDA devices (1.421s)
2022-05-18T03:45:18.9008098Z 
2022-05-18T03:45:18.9008539Z ----------------------------------------------------------------------
2022-05-18T03:45:18.9008946Z Ran 1 test in 1.422s
2022-05-18T03:45:18.9009114Z 
2022-05-18T03:45:18.9009222Z OK (skipped=1)
2022-05-18T03:45:18.9009388Z 
2022-05-18T03:45:18.9009515Z Generating XML reports...
2022-05-18T03:45:18.9042302Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518034517.xml
2022-05-18T03:45:19.6470233Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj8912fe_
2022-05-18T03:45:19.6471302Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj8912fe_/_remote_module_non_scriptable.py
2022-05-18T03:45:19.8937981Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T03:45:19.8947319Z 
2022-05-18T03:45:19.8947453Z Running tests...
2022-05-18T03:45:19.8947848Z ----------------------------------------------------------------------
2022-05-18T03:45:20.2066982Z   test_device_map_gpu_mixed_4 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31090
2022-05-18T03:45:20.2088875Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31091
2022-05-18T03:45:20.2112464Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31092
2022-05-18T03:45:20.2135611Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 31093
2022-05-18T03:45:20.8066281Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9u4kx3mb
2022-05-18T03:45:20.8067478Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9u4kx3mb/_remote_module_non_scriptable.py
2022-05-18T03:45:20.8199388Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl6iekuf6
2022-05-18T03:45:20.8200849Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl6iekuf6/_remote_module_non_scriptable.py
2022-05-18T03:45:20.8413321Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppnl1gaii
2022-05-18T03:45:20.8414843Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppnl1gaii/_remote_module_non_scriptable.py
2022-05-18T03:45:20.8700896Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy7f6m6ky
2022-05-18T03:45:20.8701840Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy7f6m6ky/_remote_module_non_scriptable.py
2022-05-18T03:45:21.0570768Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:45:21.0683078Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:45:21.0929367Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:45:21.1187787Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:45:21.3170691Z skip: Need at least 2 CUDA devices (1.422s)
2022-05-18T03:45:21.3170953Z 
2022-05-18T03:45:21.3171406Z ----------------------------------------------------------------------
2022-05-18T03:45:21.3171826Z Ran 1 test in 1.422s
2022-05-18T03:45:21.3172001Z 
2022-05-18T03:45:21.3172098Z OK (skipped=1)
2022-05-18T03:45:21.3172271Z 
2022-05-18T03:45:21.3172405Z Generating XML reports...
2022-05-18T03:45:21.3205338Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518034519.xml
2022-05-18T03:45:22.0754416Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpknxeau_3
2022-05-18T03:45:22.0755261Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpknxeau_3/_remote_module_non_scriptable.py
2022-05-18T03:45:22.3261796Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T03:45:22.3271067Z 
2022-05-18T03:45:22.3271221Z Running tests...
2022-05-18T03:45:22.3271841Z ----------------------------------------------------------------------
2022-05-18T03:45:22.6380428Z   test_device_map_gpu_mixed_5 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31145
2022-05-18T03:45:22.6402153Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31146
2022-05-18T03:45:22.6424884Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31147
2022-05-18T03:45:22.6448444Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 31148
2022-05-18T03:45:23.2978420Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_evinthy
2022-05-18T03:45:23.2979221Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_evinthy/_remote_module_non_scriptable.py
2022-05-18T03:45:23.3160390Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_ihj38qr
2022-05-18T03:45:23.3161174Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_ihj38qr/_remote_module_non_scriptable.py
2022-05-18T03:45:23.3524366Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxcjht77_
2022-05-18T03:45:23.3525137Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxcjht77_/_remote_module_non_scriptable.py
2022-05-18T03:45:23.3722805Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdnl8p368
2022-05-18T03:45:23.3723770Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdnl8p368/_remote_module_non_scriptable.py
2022-05-18T03:45:23.5476739Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:45:23.5668397Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:45:23.6039814Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:45:23.6241344Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:45:23.8485631Z skip: Need at least 2 CUDA devices (1.521s)
2022-05-18T03:45:23.8485914Z 
2022-05-18T03:45:23.8486309Z ----------------------------------------------------------------------
2022-05-18T03:45:23.8486567Z Ran 1 test in 1.521s
2022-05-18T03:45:23.8486668Z 
2022-05-18T03:45:23.8486746Z OK (skipped=1)
2022-05-18T03:45:23.8486870Z 
2022-05-18T03:45:23.8486957Z Generating XML reports...
2022-05-18T03:45:23.8518452Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518034522.xml
2022-05-18T03:45:24.6102064Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppvi2ddcq
2022-05-18T03:45:24.6103109Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppvi2ddcq/_remote_module_non_scriptable.py
2022-05-18T03:45:24.8578142Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T03:45:24.8588210Z 
2022-05-18T03:45:24.8588330Z Running tests...
2022-05-18T03:45:24.8589049Z ----------------------------------------------------------------------
2022-05-18T03:45:25.1754664Z   test_device_map_gpu_mixed_6 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31200
2022-05-18T03:45:25.1775895Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31201
2022-05-18T03:45:25.1799424Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31202
2022-05-18T03:45:25.1823381Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 31203
2022-05-18T03:45:25.7661770Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj6b8mp96
2022-05-18T03:45:25.7662560Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj6b8mp96/_remote_module_non_scriptable.py
2022-05-18T03:45:25.7744283Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpga5elg4t
2022-05-18T03:45:25.7745165Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpga5elg4t/_remote_module_non_scriptable.py
2022-05-18T03:45:25.8155816Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr0gryxnq
2022-05-18T03:45:25.8156898Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr0gryxnq/_remote_module_non_scriptable.py
2022-05-18T03:45:25.8243426Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1etxy47_
2022-05-18T03:45:25.8244639Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1etxy47_/_remote_module_non_scriptable.py
2022-05-18T03:45:26.0168332Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:45:26.0218218Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:45:26.0755489Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:45:26.0780183Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:45:26.2858769Z skip: Need at least 2 CUDA devices (1.427s)
2022-05-18T03:45:26.2859189Z 
2022-05-18T03:45:26.2859962Z ----------------------------------------------------------------------
2022-05-18T03:45:26.2860244Z Ran 1 test in 1.427s
2022-05-18T03:45:26.2860382Z 
2022-05-18T03:45:26.2860447Z OK (skipped=1)
2022-05-18T03:45:26.2860563Z 
2022-05-18T03:45:26.2860657Z Generating XML reports...
2022-05-18T03:45:26.2892439Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518034524.xml
2022-05-18T03:45:27.0448080Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppdqiokpo
2022-05-18T03:45:27.0449659Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppdqiokpo/_remote_module_non_scriptable.py
2022-05-18T03:45:27.2946417Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T03:45:27.2956155Z 
2022-05-18T03:45:27.2956258Z Running tests...
2022-05-18T03:45:27.2956739Z ----------------------------------------------------------------------
2022-05-18T03:45:27.6145758Z   test_device_map_gpu_mixed_7 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31255
2022-05-18T03:45:27.6168534Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31256
2022-05-18T03:45:27.6191471Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31257
2022-05-18T03:45:27.6214649Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 31258
2022-05-18T03:45:28.1963524Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_qax5kgr
2022-05-18T03:45:28.1964288Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_qax5kgr/_remote_module_non_scriptable.py
2022-05-18T03:45:28.2157757Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3ndho2x_
2022-05-18T03:45:28.2159086Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3ndho2x_/_remote_module_non_scriptable.py
2022-05-18T03:45:28.2596598Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpko1f1cvj
2022-05-18T03:45:28.2597345Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpko1f1cvj/_remote_module_non_scriptable.py
2022-05-18T03:45:28.2632294Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvgqsi869
2022-05-18T03:45:28.2633799Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvgqsi869/_remote_module_non_scriptable.py
2022-05-18T03:45:28.4464303Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:45:28.4683306Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:45:28.5107398Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:45:28.5161064Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:45:28.7249940Z skip: Need at least 2 CUDA devices (1.429s)
2022-05-18T03:45:28.7250137Z 
2022-05-18T03:45:28.7250444Z ----------------------------------------------------------------------
2022-05-18T03:45:28.7250946Z Ran 1 test in 1.429s
2022-05-18T03:45:28.7251061Z 
2022-05-18T03:45:28.7251122Z OK (skipped=1)
2022-05-18T03:45:28.7251234Z 
2022-05-18T03:45:28.7251386Z Generating XML reports...
2022-05-18T03:45:28.7283461Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518034527.xml
2022-05-18T03:45:29.4917782Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyye2363v
2022-05-18T03:45:29.4918472Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyye2363v/_remote_module_non_scriptable.py
2022-05-18T03:45:29.7399456Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T03:45:29.7409637Z 
2022-05-18T03:45:29.7409736Z Running tests...
2022-05-18T03:45:29.7410672Z ----------------------------------------------------------------------
2022-05-18T03:45:30.0629443Z   test_device_map_gpu_mixed_8 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31310
2022-05-18T03:45:30.0652403Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31311
2022-05-18T03:45:30.0675576Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31312
2022-05-18T03:45:30.0700862Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 31313
2022-05-18T03:45:30.6816074Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmkixgoa1
2022-05-18T03:45:30.6818644Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmkixgoa1/_remote_module_non_scriptable.py
2022-05-18T03:45:30.6911301Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqqgtoxu4
2022-05-18T03:45:30.6912499Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqqgtoxu4/_remote_module_non_scriptable.py
2022-05-18T03:45:30.7156559Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgjg8lzy0
2022-05-18T03:45:30.7157774Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgjg8lzy0/_remote_module_non_scriptable.py
2022-05-18T03:45:30.7298042Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxfyxpn0o
2022-05-18T03:45:30.7298994Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxfyxpn0o/_remote_module_non_scriptable.py
2022-05-18T03:45:30.9311555Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:45:30.9459130Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:45:30.9649692Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:45:30.9821891Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:45:31.1736677Z skip: Need at least 2 CUDA devices (1.432s)
2022-05-18T03:45:31.1737123Z 
2022-05-18T03:45:31.1737635Z ----------------------------------------------------------------------
2022-05-18T03:45:31.1738063Z Ran 1 test in 1.433s
2022-05-18T03:45:31.1738196Z 
2022-05-18T03:45:31.1738279Z OK (skipped=1)
2022-05-18T03:45:31.1738390Z 
2022-05-18T03:45:31.1738480Z Generating XML reports...
2022-05-18T03:45:31.1771914Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518034529.xml
2022-05-18T03:45:31.9487576Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsi9r8_uc
2022-05-18T03:45:31.9488136Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsi9r8_uc/_remote_module_non_scriptable.py
2022-05-18T03:45:32.1965525Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T03:45:32.1975324Z 
2022-05-18T03:45:32.1975574Z Running tests...
2022-05-18T03:45:32.1975999Z ----------------------------------------------------------------------
2022-05-18T03:45:32.5185943Z   test_device_map_gpu_mixed_self_1 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31365
2022-05-18T03:45:32.5206730Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31366
2022-05-18T03:45:32.5229745Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31367
2022-05-18T03:45:32.5254152Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 31368
2022-05-18T03:45:33.1372954Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp396fobja
2022-05-18T03:45:33.1374515Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp396fobja/_remote_module_non_scriptable.py
2022-05-18T03:45:33.1417138Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpidxq_fx7
2022-05-18T03:45:33.1418881Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpidxq_fx7/_remote_module_non_scriptable.py
2022-05-18T03:45:33.1537803Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq5mjmams
2022-05-18T03:45:33.1539404Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq5mjmams/_remote_module_non_scriptable.py
2022-05-18T03:45:33.1574669Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9bkeuhi9
2022-05-18T03:45:33.1576058Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9bkeuhi9/_remote_module_non_scriptable.py
2022-05-18T03:45:33.3871563Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:45:33.3890816Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:45:33.4042042Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:45:33.6288633Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:45:33.6289159Z skip: Need at least 2 CUDA devices (1.431s)
2022-05-18T03:45:33.6289301Z 
2022-05-18T03:45:33.6289613Z ----------------------------------------------------------------------
2022-05-18T03:45:33.6289871Z Ran 1 test in 1.431s
2022-05-18T03:45:33.6289987Z 
2022-05-18T03:45:33.6290063Z OK (skipped=1)
2022-05-18T03:45:33.6290156Z 
2022-05-18T03:45:33.6290245Z Generating XML reports...
2022-05-18T03:45:33.6322203Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518034532.xml
2022-05-18T03:45:34.3822244Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkiarrh2u
2022-05-18T03:45:34.3823423Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkiarrh2u/_remote_module_non_scriptable.py
2022-05-18T03:45:34.6293741Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T03:45:34.6303806Z 
2022-05-18T03:45:34.6303943Z Running tests...
2022-05-18T03:45:34.6304976Z ----------------------------------------------------------------------
2022-05-18T03:45:34.9442849Z   test_device_map_gpu_mixed_self_2 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31420
2022-05-18T03:45:34.9465200Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31421
2022-05-18T03:45:34.9487745Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31422
2022-05-18T03:45:34.9510956Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 31423
2022-05-18T03:45:35.5434472Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwv741fpk
2022-05-18T03:45:35.5435757Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwv741fpk/_remote_module_non_scriptable.py
2022-05-18T03:45:35.5627818Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8_nnm0w7
2022-05-18T03:45:35.5629209Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8_nnm0w7/_remote_module_non_scriptable.py
2022-05-18T03:45:35.5849679Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0zsr7_fl
2022-05-18T03:45:35.5850438Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0zsr7_fl/_remote_module_non_scriptable.py
2022-05-18T03:45:35.5963452Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0faqrm2a
2022-05-18T03:45:35.5964611Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0faqrm2a/_remote_module_non_scriptable.py
2022-05-18T03:45:35.7945743Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:45:35.8121437Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:45:35.8367203Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:45:35.8527028Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:45:36.0546990Z skip: Need at least 2 CUDA devices (1.424s)
2022-05-18T03:45:36.0547370Z 
2022-05-18T03:45:36.0547868Z ----------------------------------------------------------------------
2022-05-18T03:45:36.0548121Z Ran 1 test in 1.424s
2022-05-18T03:45:36.0548241Z 
2022-05-18T03:45:36.0548315Z OK (skipped=1)
2022-05-18T03:45:36.0548409Z 
2022-05-18T03:45:36.0548495Z Generating XML reports...
2022-05-18T03:45:36.0579454Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518034534.xml
2022-05-18T03:45:36.8051757Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpecampd54
2022-05-18T03:45:36.8052559Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpecampd54/_remote_module_non_scriptable.py
2022-05-18T03:45:37.0511215Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T03:45:37.0521142Z 
2022-05-18T03:45:37.0521227Z Running tests...
2022-05-18T03:45:37.0522185Z ----------------------------------------------------------------------
2022-05-18T03:45:37.3662641Z   test_device_map_gpu_mixed_self_3 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31475
2022-05-18T03:45:37.3685193Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31476
2022-05-18T03:45:37.3708677Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31477
2022-05-18T03:45:37.3732848Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 31478
2022-05-18T03:45:37.9609356Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzw8jhtvd
2022-05-18T03:45:37.9610152Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzw8jhtvd/_remote_module_non_scriptable.py
2022-05-18T03:45:37.9760316Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpno8612m4
2022-05-18T03:45:37.9761095Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpno8612m4/_remote_module_non_scriptable.py
2022-05-18T03:45:37.9972446Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpie_bm2_w
2022-05-18T03:45:37.9973330Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpie_bm2_w/_remote_module_non_scriptable.py
2022-05-18T03:45:38.0017747Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9ofmzekk
2022-05-18T03:45:38.0019470Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9ofmzekk/_remote_module_non_scriptable.py
2022-05-18T03:45:38.2079223Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:45:38.2243485Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:45:38.2481984Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:45:38.2514099Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:45:38.4769168Z skip: Need at least 2 CUDA devices (1.425s)
2022-05-18T03:45:38.4769638Z 
2022-05-18T03:45:38.4770258Z ----------------------------------------------------------------------
2022-05-18T03:45:38.4770699Z Ran 1 test in 1.425s
2022-05-18T03:45:38.4770901Z 
2022-05-18T03:45:38.4771006Z OK (skipped=1)
2022-05-18T03:45:38.4771170Z 
2022-05-18T03:45:38.4771305Z Generating XML reports...
2022-05-18T03:45:38.4803850Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518034537.xml
2022-05-18T03:45:39.2204932Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm6ocv98l
2022-05-18T03:45:39.2205878Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm6ocv98l/_remote_module_non_scriptable.py
2022-05-18T03:45:39.4667462Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T03:45:39.4676220Z 
2022-05-18T03:45:39.4676348Z Running tests...
2022-05-18T03:45:39.4676942Z ----------------------------------------------------------------------
2022-05-18T03:45:39.7807480Z   test_device_map_gpu_mixed_self_4 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31530
2022-05-18T03:45:39.7830064Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31531
2022-05-18T03:45:39.7852402Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31532
2022-05-18T03:45:39.7876248Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 31533
2022-05-18T03:45:40.3827685Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppydvfq0j
2022-05-18T03:45:40.3828410Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppydvfq0j/_remote_module_non_scriptable.py
2022-05-18T03:45:40.4000103Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqlssxhib
2022-05-18T03:45:40.4001351Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqlssxhib/_remote_module_non_scriptable.py
2022-05-18T03:45:40.4239463Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_hl3l494
2022-05-18T03:45:40.4240352Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_hl3l494/_remote_module_non_scriptable.py
2022-05-18T03:45:40.4470509Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp62wymj7w
2022-05-18T03:45:40.4471606Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp62wymj7w/_remote_module_non_scriptable.py
2022-05-18T03:45:40.6296978Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:45:40.6538194Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:45:40.6723338Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:45:40.6988116Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:45:40.8912092Z skip: Need at least 2 CUDA devices (1.423s)
2022-05-18T03:45:40.8912398Z 
2022-05-18T03:45:40.8912891Z ----------------------------------------------------------------------
2022-05-18T03:45:40.8913167Z Ran 1 test in 1.423s
2022-05-18T03:45:40.8913289Z 
2022-05-18T03:45:40.8913363Z OK (skipped=1)
2022-05-18T03:45:40.8913470Z 
2022-05-18T03:45:40.8913555Z Generating XML reports...
2022-05-18T03:45:40.8944796Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518034539.xml
2022-05-18T03:45:41.6336519Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdlbjd_xy
2022-05-18T03:45:41.6337563Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdlbjd_xy/_remote_module_non_scriptable.py
2022-05-18T03:45:41.8804496Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T03:45:41.8813673Z 
2022-05-18T03:45:41.8813772Z Running tests...
2022-05-18T03:45:41.8814198Z ----------------------------------------------------------------------
2022-05-18T03:45:42.1923889Z   test_device_map_gpu_mixed_self_5 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31585
2022-05-18T03:45:42.1947083Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31586
2022-05-18T03:45:42.1969422Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31587
2022-05-18T03:45:42.1993433Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 31588
2022-05-18T03:45:42.8105825Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgguc9i9v
2022-05-18T03:45:42.8107034Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgguc9i9v/_remote_module_non_scriptable.py
2022-05-18T03:45:42.8147494Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphhhyf99a
2022-05-18T03:45:42.8148830Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphhhyf99a/_remote_module_non_scriptable.py
2022-05-18T03:45:42.8420692Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3hw1ksz7
2022-05-18T03:45:42.8431882Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3hw1ksz7/_remote_module_non_scriptable.py
2022-05-18T03:45:42.8580976Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbkk1l_2t
2022-05-18T03:45:42.8583204Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbkk1l_2t/_remote_module_non_scriptable.py
2022-05-18T03:45:43.0611962Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:45:43.0612976Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:45:43.1053464Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:45:43.1064341Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:45:43.3029519Z skip: Need at least 2 CUDA devices (1.421s)
2022-05-18T03:45:43.3029766Z 
2022-05-18T03:45:43.3030222Z ----------------------------------------------------------------------
2022-05-18T03:45:43.3030617Z Ran 1 test in 1.421s
2022-05-18T03:45:43.3030807Z 
2022-05-18T03:45:43.3030920Z OK (skipped=1)
2022-05-18T03:45:43.3031074Z 
2022-05-18T03:45:43.3031218Z Generating XML reports...
2022-05-18T03:45:43.3063999Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518034541.xml
2022-05-18T03:45:44.0463906Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeed94na1
2022-05-18T03:45:44.0464416Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeed94na1/_remote_module_non_scriptable.py
2022-05-18T03:45:44.2932157Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T03:45:44.2941218Z 
2022-05-18T03:45:44.2941317Z Running tests...
2022-05-18T03:45:44.2942184Z ----------------------------------------------------------------------
2022-05-18T03:45:44.6060555Z   test_device_map_gpu_mixed_self_6 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31640
2022-05-18T03:45:44.6082786Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31641
2022-05-18T03:45:44.6105706Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31642
2022-05-18T03:45:44.6129046Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 31643
2022-05-18T03:45:45.2313728Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0z5zxodv
2022-05-18T03:45:45.2314763Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0z5zxodv/_remote_module_non_scriptable.py
2022-05-18T03:45:45.2618413Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgqj98qz4
2022-05-18T03:45:45.2619083Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp71hkx_l8
2022-05-18T03:45:45.2619788Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgqj98qz4/_remote_module_non_scriptable.py
2022-05-18T03:45:45.2621033Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp71hkx_l8/_remote_module_non_scriptable.py
2022-05-18T03:45:45.2642320Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3pfe_dec
2022-05-18T03:45:45.2643940Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3pfe_dec/_remote_module_non_scriptable.py
2022-05-18T03:45:45.4828851Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:45:45.5105388Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:45:45.5135806Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:45:45.5191885Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:45:45.7164332Z skip: Need at least 2 CUDA devices (1.422s)
2022-05-18T03:45:45.7164710Z 
2022-05-18T03:45:45.7165020Z ----------------------------------------------------------------------
2022-05-18T03:45:45.7165285Z Ran 1 test in 1.422s
2022-05-18T03:45:45.7165401Z 
2022-05-18T03:45:45.7165474Z OK (skipped=1)
2022-05-18T03:45:45.7165569Z 
2022-05-18T03:45:45.7165657Z Generating XML reports...
2022-05-18T03:45:45.7197123Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518034544.xml
2022-05-18T03:45:46.4597892Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjg8axwea
2022-05-18T03:45:46.4598808Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjg8axwea/_remote_module_non_scriptable.py
2022-05-18T03:45:46.7061701Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T03:45:46.7070630Z 
2022-05-18T03:45:46.7070720Z Running tests...
2022-05-18T03:45:46.7071116Z ----------------------------------------------------------------------
2022-05-18T03:45:47.0186587Z   test_device_map_gpu_mixed_self_7 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31695
2022-05-18T03:45:47.0208868Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31696
2022-05-18T03:45:47.0231332Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31697
2022-05-18T03:45:47.0254726Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 31698
2022-05-18T03:45:47.6425160Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmw1xoz_n
2022-05-18T03:45:47.6425990Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmw1xoz_n/_remote_module_non_scriptable.py
2022-05-18T03:45:47.6500863Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpczr1vxkh
2022-05-18T03:45:47.6502348Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpczr1vxkh/_remote_module_non_scriptable.py
2022-05-18T03:45:47.6530302Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyjtqwdiu
2022-05-18T03:45:47.6532236Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyjtqwdiu/_remote_module_non_scriptable.py
2022-05-18T03:45:47.6595566Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6i6hpict
2022-05-18T03:45:47.6597174Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6i6hpict/_remote_module_non_scriptable.py
2022-05-18T03:45:47.8903313Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:45:47.8996528Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:45:47.9036941Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:45:47.9078933Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:45:48.1292646Z skip: Need at least 2 CUDA devices (1.422s)
2022-05-18T03:45:48.1292902Z 
2022-05-18T03:45:48.1293354Z ----------------------------------------------------------------------
2022-05-18T03:45:48.1293748Z Ran 1 test in 1.422s
2022-05-18T03:45:48.1293928Z 
2022-05-18T03:45:48.1294035Z OK (skipped=1)
2022-05-18T03:45:48.1294204Z 
2022-05-18T03:45:48.1294342Z Generating XML reports...
2022-05-18T03:45:48.1327240Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518034546.xml
2022-05-18T03:45:48.8767940Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6ypni6p_
2022-05-18T03:45:48.8768673Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6ypni6p_/_remote_module_non_scriptable.py
2022-05-18T03:45:49.1245667Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T03:45:49.1255228Z 
2022-05-18T03:45:49.1255509Z Running tests...
2022-05-18T03:45:49.1256137Z ----------------------------------------------------------------------
2022-05-18T03:45:49.4402108Z   test_device_map_gpu_mixed_self_8 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31750
2022-05-18T03:45:49.4423465Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31751
2022-05-18T03:45:49.4446048Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31752
2022-05-18T03:45:49.4469734Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 31753
2022-05-18T03:45:50.0678012Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0nr1ncad
2022-05-18T03:45:50.0678829Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0nr1ncad/_remote_module_non_scriptable.py
2022-05-18T03:45:50.0715405Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq520pmv7
2022-05-18T03:45:50.0716978Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq520pmv7/_remote_module_non_scriptable.py
2022-05-18T03:45:50.0802744Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2w87cnzd
2022-05-18T03:45:50.0804517Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2w87cnzd/_remote_module_non_scriptable.py
2022-05-18T03:45:50.0972903Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf8q88gdx
2022-05-18T03:45:50.0974176Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf8q88gdx/_remote_module_non_scriptable.py
2022-05-18T03:45:50.3161641Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:45:50.3210061Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:45:50.3255980Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:45:50.3547439Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:45:50.5506379Z skip: Need at least 2 CUDA devices (1.425s)
2022-05-18T03:45:50.5506558Z 
2022-05-18T03:45:50.5506850Z ----------------------------------------------------------------------
2022-05-18T03:45:50.5507122Z Ran 1 test in 1.425s
2022-05-18T03:45:50.5507235Z 
2022-05-18T03:45:50.5507308Z OK (skipped=1)
2022-05-18T03:45:50.5507671Z 
2022-05-18T03:45:50.5507758Z Generating XML reports...
2022-05-18T03:45:50.5539430Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518034549.xml
2022-05-18T03:45:51.2981931Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy5pfflfb
2022-05-18T03:45:51.2982617Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy5pfflfb/_remote_module_non_scriptable.py
2022-05-18T03:45:51.5455215Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T03:45:51.5465629Z 
2022-05-18T03:45:51.5465785Z Running tests...
2022-05-18T03:45:51.5466231Z ----------------------------------------------------------------------
2022-05-18T03:45:51.8567525Z   test_device_map_gpu_non_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31805
2022-05-18T03:45:51.8590011Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31806
2022-05-18T03:45:51.8612784Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31807
2022-05-18T03:45:51.8636770Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 31808
2022-05-18T03:45:52.4964109Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpodkg_n22
2022-05-18T03:45:52.4964885Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpodkg_n22/_remote_module_non_scriptable.py
2022-05-18T03:45:52.5042455Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4avk_27b
2022-05-18T03:45:52.5043657Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4avk_27b/_remote_module_non_scriptable.py
2022-05-18T03:45:52.5152483Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg8v3ucrs
2022-05-18T03:45:52.5153569Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg8v3ucrs/_remote_module_non_scriptable.py
2022-05-18T03:45:52.5282345Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv6uo12a_
2022-05-18T03:45:52.5283709Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv6uo12a_/_remote_module_non_scriptable.py
2022-05-18T03:45:52.7444980Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:45:52.7552953Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:45:52.7633740Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:45:52.7768999Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:45:52.9672113Z skip: Need at least 2 CUDA devices (1.420s)
2022-05-18T03:45:52.9672386Z 
2022-05-18T03:45:52.9672865Z ----------------------------------------------------------------------
2022-05-18T03:45:52.9673342Z Ran 1 test in 1.421s
2022-05-18T03:45:52.9673544Z 
2022-05-18T03:45:52.9673618Z OK (skipped=1)
2022-05-18T03:45:52.9673726Z 
2022-05-18T03:45:52.9673798Z Generating XML reports...
2022-05-18T03:45:52.9706196Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518034551.xml
2022-05-18T03:45:53.7143881Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkiw627tx
2022-05-18T03:45:53.7144882Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkiw627tx/_remote_module_non_scriptable.py
2022-05-18T03:45:53.9594647Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T03:45:53.9603729Z 
2022-05-18T03:45:53.9603853Z Running tests...
2022-05-18T03:45:53.9604490Z ----------------------------------------------------------------------
2022-05-18T03:45:54.2706934Z   test_device_map_gpu_non_default_to_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31860
2022-05-18T03:45:54.2729221Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31861
2022-05-18T03:45:54.2752090Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31862
2022-05-18T03:45:54.2776363Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 31863
2022-05-18T03:45:54.8868190Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps9wxhpbg
2022-05-18T03:45:54.8868899Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps9wxhpbg/_remote_module_non_scriptable.py
2022-05-18T03:45:54.8955130Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl0xaga9b
2022-05-18T03:45:54.8956342Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl0xaga9b/_remote_module_non_scriptable.py
2022-05-18T03:45:54.9088100Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnkhyb5v5
2022-05-18T03:45:54.9090085Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnkhyb5v5/_remote_module_non_scriptable.py
2022-05-18T03:45:54.9138828Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk915mf0g
2022-05-18T03:45:54.9140894Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk915mf0g/_remote_module_non_scriptable.py
2022-05-18T03:45:55.1348089Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:45:55.1429551Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:45:55.1572167Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:45:55.1607234Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:45:55.3810676Z skip: Need at least 2 CUDA devices (1.420s)
2022-05-18T03:45:55.3811021Z 
2022-05-18T03:45:55.3811468Z ----------------------------------------------------------------------
2022-05-18T03:45:55.3811761Z Ran 1 test in 1.421s
2022-05-18T03:45:55.3811876Z 
2022-05-18T03:45:55.3811951Z OK (skipped=1)
2022-05-18T03:45:55.3812045Z 
2022-05-18T03:45:55.3812140Z Generating XML reports...
2022-05-18T03:45:55.3844866Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518034553.xml
2022-05-18T03:45:56.1368008Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpphxf7uy6
2022-05-18T03:45:56.1368648Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpphxf7uy6/_remote_module_non_scriptable.py
2022-05-18T03:45:56.3846608Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T03:45:56.3856260Z 
2022-05-18T03:45:56.3856346Z Running tests...
2022-05-18T03:45:56.3857117Z ----------------------------------------------------------------------
2022-05-18T03:45:56.7004225Z   test_device_map_gpu_to_cpu_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31915
2022-05-18T03:45:56.7026872Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31916
2022-05-18T03:45:56.7050399Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31917
2022-05-18T03:45:56.7074349Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 31918
2022-05-18T03:45:57.3515473Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvz5e11xz
2022-05-18T03:45:57.3516441Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvz5e11xz/_remote_module_non_scriptable.py
2022-05-18T03:45:57.3895121Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpohvw830s
2022-05-18T03:45:57.3896305Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpohvw830s/_remote_module_non_scriptable.py
2022-05-18T03:45:57.4020493Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdx5o163m
2022-05-18T03:45:57.4022633Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdx5o163m/_remote_module_non_scriptable.py
2022-05-18T03:45:57.4183062Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7x7z2ebl
2022-05-18T03:45:57.4184595Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7x7z2ebl/_remote_module_non_scriptable.py
2022-05-18T03:45:57.6006879Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:45:57.6384466Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:45:57.6523413Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:45:57.6775246Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:45:57.8109441Z skip: Need at least 1 CUDA device (1.425s)
2022-05-18T03:45:57.8109680Z 
2022-05-18T03:45:57.8109981Z ----------------------------------------------------------------------
2022-05-18T03:45:57.8110222Z Ran 1 test in 1.425s
2022-05-18T03:45:57.8110344Z 
2022-05-18T03:45:57.8110419Z OK (skipped=1)
2022-05-18T03:45:57.8110527Z 
2022-05-18T03:45:57.8110616Z Generating XML reports...
2022-05-18T03:45:57.8144276Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518034556.xml
2022-05-18T03:45:58.5542631Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3p61v5p1
2022-05-18T03:45:58.5543659Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3p61v5p1/_remote_module_non_scriptable.py
2022-05-18T03:45:58.8013966Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T03:45:58.8024213Z 
2022-05-18T03:45:58.8024496Z Running tests...
2022-05-18T03:45:59.1126504Z ----------------------------------------------------------------------
2022-05-18T03:45:59.1127371Z   test_device_map_gpu_to_cpu_non_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31970
2022-05-18T03:45:59.1147742Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31971
2022-05-18T03:45:59.1170198Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31972
2022-05-18T03:45:59.1193887Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 31973
2022-05-18T03:45:59.8201481Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5ee9uehk
2022-05-18T03:45:59.8202248Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5ee9uehk/_remote_module_non_scriptable.py
2022-05-18T03:45:59.8275087Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl1_ecgk5
2022-05-18T03:45:59.8276281Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl1_ecgk5/_remote_module_non_scriptable.py
2022-05-18T03:45:59.8529617Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj5_rgz_8
2022-05-18T03:45:59.8530692Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj5_rgz_8/_remote_module_non_scriptable.py
2022-05-18T03:45:59.8608393Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp10h8lfck
2022-05-18T03:45:59.8609380Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp10h8lfck/_remote_module_non_scriptable.py
2022-05-18T03:46:00.0717137Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:46:00.0750258Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:46:00.1079527Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:46:00.1166149Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:46:00.3231271Z skip: Need at least 2 CUDA devices (1.520s)
2022-05-18T03:46:00.3231623Z 
2022-05-18T03:46:00.3232326Z ----------------------------------------------------------------------
2022-05-18T03:46:00.3232586Z Ran 1 test in 1.521s
2022-05-18T03:46:00.3232703Z 
2022-05-18T03:46:00.3232777Z OK (skipped=1)
2022-05-18T03:46:00.3232887Z 
2022-05-18T03:46:00.3232959Z Generating XML reports...
2022-05-18T03:46:00.3264060Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518034558.xml
2022-05-18T03:46:01.0719463Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkcoearyy
2022-05-18T03:46:01.0719935Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkcoearyy/_remote_module_non_scriptable.py
2022-05-18T03:46:01.3202783Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T03:46:01.3212629Z 
2022-05-18T03:46:01.3212760Z Running tests...
2022-05-18T03:46:01.3213335Z ----------------------------------------------------------------------
2022-05-18T03:46:01.6347252Z   test_device_maps_gpu (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32025
2022-05-18T03:46:01.6370875Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32026
2022-05-18T03:46:01.6394222Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 32027
2022-05-18T03:46:01.6417905Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 32028
2022-05-18T03:46:02.2920967Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_9ougwwl
2022-05-18T03:46:02.2921721Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_9ougwwl/_remote_module_non_scriptable.py
2022-05-18T03:46:02.2932301Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp90nu1toe
2022-05-18T03:46:02.2933367Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp90nu1toe/_remote_module_non_scriptable.py
2022-05-18T03:46:02.3052462Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5pc_ms6n
2022-05-18T03:46:02.3053473Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5pc_ms6n/_remote_module_non_scriptable.py
2022-05-18T03:46:02.3158095Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1k8z19i4
2022-05-18T03:46:02.3158892Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1k8z19i4/_remote_module_non_scriptable.py
2022-05-18T03:46:02.5412812Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:46:02.5419017Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:46:02.5540336Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:46:02.5668141Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:46:02.7452359Z skip: Need at least 2 CUDA devices (1.424s)
2022-05-18T03:46:02.7452666Z 
2022-05-18T03:46:02.7453200Z ----------------------------------------------------------------------
2022-05-18T03:46:02.7453631Z Ran 1 test in 1.424s
2022-05-18T03:46:02.7453747Z 
2022-05-18T03:46:02.7453808Z OK (skipped=1)
2022-05-18T03:46:02.7453914Z 
2022-05-18T03:46:02.7454224Z Generating XML reports...
2022-05-18T03:46:02.7487323Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518034601.xml
2022-05-18T03:46:03.4871745Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptftdzk9k
2022-05-18T03:46:03.4872693Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptftdzk9k/_remote_module_non_scriptable.py
2022-05-18T03:46:03.7337073Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T03:46:03.7347099Z 
2022-05-18T03:46:03.7347201Z Running tests...
2022-05-18T03:46:03.7347931Z ----------------------------------------------------------------------
2022-05-18T03:46:04.0471785Z   test_device_maps_in_options (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32080
2022-05-18T03:46:04.0493666Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32081
2022-05-18T03:46:04.0516480Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 32082
2022-05-18T03:46:04.0540649Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 32083
2022-05-18T03:46:04.6578912Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxcebqmjn
2022-05-18T03:46:04.6579684Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxcebqmjn/_remote_module_non_scriptable.py
2022-05-18T03:46:04.6834565Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcwmgtzsn
2022-05-18T03:46:04.6835868Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcwmgtzsn/_remote_module_non_scriptable.py
2022-05-18T03:46:04.6944519Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp286sq_t0
2022-05-18T03:46:04.6945924Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp286sq_t0/_remote_module_non_scriptable.py
2022-05-18T03:46:04.7053083Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuhh2qv_2
2022-05-18T03:46:04.7054397Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuhh2qv_2/_remote_module_non_scriptable.py
2022-05-18T03:46:04.9048575Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:46:04.9429855Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:46:04.9433416Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:46:04.9528076Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:46:05.1576276Z skip: Need at least 2 CUDA devices (1.423s)
2022-05-18T03:46:05.1576589Z 
2022-05-18T03:46:05.1577106Z ----------------------------------------------------------------------
2022-05-18T03:46:05.1577506Z Ran 1 test in 1.423s
2022-05-18T03:46:05.1577622Z 
2022-05-18T03:46:05.1577699Z OK (skipped=1)
2022-05-18T03:46:05.1577794Z 
2022-05-18T03:46:05.1577881Z Generating XML reports...
2022-05-18T03:46:05.1609593Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518034603.xml
2022-05-18T03:46:05.9054052Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp4s4zxc4
2022-05-18T03:46:05.9054925Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp4s4zxc4/_remote_module_non_scriptable.py
2022-05-18T03:46:06.1522760Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T03:46:06.1532124Z 
2022-05-18T03:46:06.1532249Z Running tests...
2022-05-18T03:46:06.1532845Z ----------------------------------------------------------------------
2022-05-18T03:46:06.4653437Z   test_device_maps_invalid_max_local_device (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32135
2022-05-18T03:46:06.4676011Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32136
2022-05-18T03:46:06.4698926Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 32137
2022-05-18T03:46:06.4722749Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 32138
2022-05-18T03:46:07.0879446Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgmsl0ucv
2022-05-18T03:46:07.0880477Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgmsl0ucv/_remote_module_non_scriptable.py
2022-05-18T03:46:07.1091138Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppk1i2zml
2022-05-18T03:46:07.1092276Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppk1i2zml/_remote_module_non_scriptable.py
2022-05-18T03:46:07.1205931Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpaz1cm1ng
2022-05-18T03:46:07.1207570Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpaz1cm1ng/_remote_module_non_scriptable.py
2022-05-18T03:46:07.1266924Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzpdl91u_
2022-05-18T03:46:07.1268834Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzpdl91u_/_remote_module_non_scriptable.py
2022-05-18T03:46:07.3382242Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:46:07.3703936Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:46:07.3724272Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:46:07.3808331Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:46:07.5757890Z skip: Need at least 1 CUDA device (1.422s)
2022-05-18T03:46:07.5758204Z 
2022-05-18T03:46:07.5758711Z ----------------------------------------------------------------------
2022-05-18T03:46:07.5759162Z Ran 1 test in 1.422s
2022-05-18T03:46:07.5759379Z 
2022-05-18T03:46:07.5759819Z OK (skipped=1)
2022-05-18T03:46:07.5760006Z 
2022-05-18T03:46:07.5760166Z Generating XML reports...
2022-05-18T03:46:07.5791222Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518034606.xml
2022-05-18T03:46:08.3226258Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpet_lmjjb
2022-05-18T03:46:08.3227054Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpet_lmjjb/_remote_module_non_scriptable.py
2022-05-18T03:46:08.5701090Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T03:46:08.5710477Z 
2022-05-18T03:46:08.5710576Z Running tests...
2022-05-18T03:46:08.5711305Z ----------------------------------------------------------------------
2022-05-18T03:46:08.8817010Z   test_device_maps_invalid_max_remote_device (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32190
2022-05-18T03:46:08.8839482Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32191
2022-05-18T03:46:08.8861718Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 32192
2022-05-18T03:46:08.8885360Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 32193
2022-05-18T03:46:09.4897047Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3kl8zgys
2022-05-18T03:46:09.4899087Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3kl8zgys/_remote_module_non_scriptable.py
2022-05-18T03:46:09.5116131Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf32w3d2x
2022-05-18T03:46:09.5116902Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf32w3d2x/_remote_module_non_scriptable.py
2022-05-18T03:46:09.5209245Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb28ev5pl
2022-05-18T03:46:09.5210513Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb28ev5pl/_remote_module_non_scriptable.py
2022-05-18T03:46:09.5226963Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmapg3m04
2022-05-18T03:46:09.5228982Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmapg3m04/_remote_module_non_scriptable.py
2022-05-18T03:46:09.7375826Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:46:09.7668616Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:46:09.7703332Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:46:09.7722997Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:46:09.9921127Z skip: Need at least 1 CUDA device (1.421s)
2022-05-18T03:46:09.9921461Z 
2022-05-18T03:46:09.9921906Z ----------------------------------------------------------------------
2022-05-18T03:46:09.9922162Z Ran 1 test in 1.421s
2022-05-18T03:46:09.9922280Z 
2022-05-18T03:46:09.9922354Z OK (skipped=1)
2022-05-18T03:46:09.9922464Z 
2022-05-18T03:46:09.9922536Z Generating XML reports...
2022-05-18T03:46:09.9953610Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518034608.xml
2022-05-18T03:46:10.7425366Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpov_lc3a5
2022-05-18T03:46:10.7426195Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpov_lc3a5/_remote_module_non_scriptable.py
2022-05-18T03:46:10.9899112Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T03:46:10.9908768Z 
2022-05-18T03:46:10.9908862Z Running tests...
2022-05-18T03:46:10.9909632Z ----------------------------------------------------------------------
2022-05-18T03:46:11.3046879Z   test_device_maps_invalid_min_device (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32245
2022-05-18T03:46:11.3067869Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32246
2022-05-18T03:46:11.3090308Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 32247
2022-05-18T03:46:11.3114367Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 32248
2022-05-18T03:46:12.0283238Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphr1jx6l8
2022-05-18T03:46:12.0284486Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphr1jx6l8/_remote_module_non_scriptable.py
2022-05-18T03:46:12.0340204Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_gqm72lp
2022-05-18T03:46:12.0341524Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_gqm72lp/_remote_module_non_scriptable.py
2022-05-18T03:46:12.0542238Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptbjgt6h7
2022-05-18T03:46:12.0543253Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptbjgt6h7/_remote_module_non_scriptable.py
2022-05-18T03:46:12.0860905Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3fn19185
2022-05-18T03:46:12.0861880Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3fn19185/_remote_module_non_scriptable.py
2022-05-18T03:46:12.2792562Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:46:12.2832732Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:46:12.3040287Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:46:12.3631883Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:46:12.5151022Z skip: Need at least 1 CUDA device (1.524s)
2022-05-18T03:46:12.5151278Z 
2022-05-18T03:46:12.5151716Z ----------------------------------------------------------------------
2022-05-18T03:46:12.5152157Z Ran 1 test in 1.524s
2022-05-18T03:46:12.5152330Z 
2022-05-18T03:46:12.5152440Z OK (skipped=1)
2022-05-18T03:46:12.5152606Z 
2022-05-18T03:46:12.5152749Z Generating XML reports...
2022-05-18T03:46:12.5185137Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518034610.xml
2022-05-18T03:46:13.2837105Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp354qx_o1
2022-05-18T03:46:13.2837831Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp354qx_o1/_remote_module_non_scriptable.py
2022-05-18T03:46:13.5325403Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T03:46:13.5334461Z 
2022-05-18T03:46:13.5334703Z Running tests...
2022-05-18T03:46:13.5335323Z ----------------------------------------------------------------------
2022-05-18T03:46:13.8497783Z   test_device_maps_many_to_one (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32300
2022-05-18T03:46:13.8520258Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32301
2022-05-18T03:46:13.8543189Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 32302
2022-05-18T03:46:13.8568629Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 32303
2022-05-18T03:46:14.4616040Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp45wsrpai
2022-05-18T03:46:14.4616842Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp45wsrpai/_remote_module_non_scriptable.py
2022-05-18T03:46:14.4743048Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp23_yznlo
2022-05-18T03:46:14.4744180Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp23_yznlo/_remote_module_non_scriptable.py
2022-05-18T03:46:14.4823978Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzgfdtclx
2022-05-18T03:46:14.4826391Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzgfdtclx/_remote_module_non_scriptable.py
2022-05-18T03:46:14.4853214Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4vw2ltpf
2022-05-18T03:46:14.4855788Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4vw2ltpf/_remote_module_non_scriptable.py
2022-05-18T03:46:14.7088423Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:46:14.7225780Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:46:14.7278408Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:46:14.7446797Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:46:14.9603400Z skip: Need at least 2 CUDA devices (1.427s)
2022-05-18T03:46:14.9603638Z 
2022-05-18T03:46:14.9604221Z ----------------------------------------------------------------------
2022-05-18T03:46:14.9604573Z Ran 1 test in 1.427s
2022-05-18T03:46:14.9604690Z 
2022-05-18T03:46:14.9604771Z OK (skipped=1)
2022-05-18T03:46:14.9604881Z 
2022-05-18T03:46:14.9604972Z Generating XML reports...
2022-05-18T03:46:14.9639336Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518034613.xml
2022-05-18T03:46:15.7093566Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw7tc4dsk
2022-05-18T03:46:15.7094045Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw7tc4dsk/_remote_module_non_scriptable.py
2022-05-18T03:46:15.9563997Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T03:46:15.9573406Z 
2022-05-18T03:46:15.9573529Z Running tests...
2022-05-18T03:46:15.9574122Z ----------------------------------------------------------------------
2022-05-18T03:46:16.2730322Z   test_device_maps_missing_config (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32355
2022-05-18T03:46:16.2752209Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32356
2022-05-18T03:46:16.2774600Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 32357
2022-05-18T03:46:16.2798650Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 32358
2022-05-18T03:46:16.8833782Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpycd5pnc_
2022-05-18T03:46:16.8834900Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpycd5pnc_/_remote_module_non_scriptable.py
2022-05-18T03:46:16.8926147Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp24b1lip8
2022-05-18T03:46:16.8928587Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp24b1lip8/_remote_module_non_scriptable.py
2022-05-18T03:46:16.9082280Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7frv4ys7
2022-05-18T03:46:16.9085836Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7frv4ys7/_remote_module_non_scriptable.py
2022-05-18T03:46:16.9092974Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgmljh_ag
2022-05-18T03:46:16.9095389Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgmljh_ag/_remote_module_non_scriptable.py
2022-05-18T03:46:17.1338544Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:46:17.1408229Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:46:17.1572674Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:46:17.1580769Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:46:17.3834828Z skip: Need at least 1 CUDA device (1.426s)
2022-05-18T03:46:17.3835145Z 
2022-05-18T03:46:17.3835501Z ----------------------------------------------------------------------
2022-05-18T03:46:17.3835738Z Ran 1 test in 1.426s
2022-05-18T03:46:17.3835850Z 
2022-05-18T03:46:17.3835944Z OK (skipped=1)
2022-05-18T03:46:17.3836052Z 
2022-05-18T03:46:17.3836137Z Generating XML reports...
2022-05-18T03:46:17.3867325Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518034615.xml
2022-05-18T03:46:18.1317107Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe0w87wyj
2022-05-18T03:46:18.1317570Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe0w87wyj/_remote_module_non_scriptable.py
2022-05-18T03:46:18.3781590Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T03:46:18.3790391Z 
2022-05-18T03:46:18.3790481Z Running tests...
2022-05-18T03:46:18.3791344Z ----------------------------------------------------------------------
2022-05-18T03:46:18.6945259Z   test_device_maps_missing_config_loop (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32410
2022-05-18T03:46:18.6967717Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32411
2022-05-18T03:46:18.6990541Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 32412
2022-05-18T03:46:18.7013848Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 32413
2022-05-18T03:46:19.3010848Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6fwvzygx
2022-05-18T03:46:19.3012998Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6fwvzygx/_remote_module_non_scriptable.py
2022-05-18T03:46:19.3214027Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzk4fp4su
2022-05-18T03:46:19.3215137Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzk4fp4su/_remote_module_non_scriptable.py
2022-05-18T03:46:19.3604394Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxsucdjf6
2022-05-18T03:46:19.3605223Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxsucdjf6/_remote_module_non_scriptable.py
2022-05-18T03:46:19.3606591Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8n639yzd
2022-05-18T03:46:19.3610575Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8n639yzd/_remote_module_non_scriptable.py
2022-05-18T03:46:19.5539573Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:46:19.5761975Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:46:19.6111652Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:46:19.6155829Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:46:19.8048593Z skip: Need at least 1 CUDA device (1.426s)
2022-05-18T03:46:19.8048910Z 
2022-05-18T03:46:19.8049404Z ----------------------------------------------------------------------
2022-05-18T03:46:19.8049687Z Ran 1 test in 1.426s
2022-05-18T03:46:19.8049803Z 
2022-05-18T03:46:19.8049877Z OK (skipped=1)
2022-05-18T03:46:19.8049987Z 
2022-05-18T03:46:19.8050075Z Generating XML reports...
2022-05-18T03:46:19.8082033Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518034618.xml
2022-05-18T03:46:20.5594888Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbgim3w_d
2022-05-18T03:46:20.5595364Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbgim3w_d/_remote_module_non_scriptable.py
2022-05-18T03:46:20.8062140Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T03:46:20.8072482Z 
2022-05-18T03:46:20.8072744Z Running tests...
2022-05-18T03:46:20.8073401Z ----------------------------------------------------------------------
2022-05-18T03:46:21.1196888Z   test_device_maps_missing_config_not_timeout (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32465
2022-05-18T03:46:21.1219174Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32466
2022-05-18T03:46:21.1241876Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 32467
2022-05-18T03:46:21.1265443Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 32468
2022-05-18T03:46:21.8059694Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzbeuq5cq
2022-05-18T03:46:21.8060911Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzbeuq5cq/_remote_module_non_scriptable.py
2022-05-18T03:46:21.8146185Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6malsmw8
2022-05-18T03:46:21.8147821Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6malsmw8/_remote_module_non_scriptable.py
2022-05-18T03:46:21.8564519Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfk6p4v37
2022-05-18T03:46:21.8565613Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfk6p4v37/_remote_module_non_scriptable.py
2022-05-18T03:46:21.8579118Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj4xmfte0
2022-05-18T03:46:21.8580987Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj4xmfte0/_remote_module_non_scriptable.py
2022-05-18T03:46:22.0551168Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:46:22.0648620Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:46:22.1058039Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:46:22.1065439Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:46:22.3303895Z skip: Need at least 1 CUDA device (1.523s)
2022-05-18T03:46:22.3304201Z 
2022-05-18T03:46:22.3304982Z ----------------------------------------------------------------------
2022-05-18T03:46:22.3305242Z Ran 1 test in 1.523s
2022-05-18T03:46:22.3305356Z 
2022-05-18T03:46:22.3305418Z OK (skipped=1)
2022-05-18T03:46:22.3305599Z 
2022-05-18T03:46:22.3305690Z Generating XML reports...
2022-05-18T03:46:22.3336395Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518034620.xml
2022-05-18T03:46:23.0704623Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3gpthsh5
2022-05-18T03:46:23.0705380Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3gpthsh5/_remote_module_non_scriptable.py
2022-05-18T03:46:23.3177910Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T03:46:23.3187677Z 
2022-05-18T03:46:23.3187782Z Running tests...
2022-05-18T03:46:23.3188746Z ----------------------------------------------------------------------
2022-05-18T03:46:23.6314700Z   test_device_maps_missing_config_remote (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32520
2022-05-18T03:46:23.6336609Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32521
2022-05-18T03:46:23.6359192Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 32522
2022-05-18T03:46:23.6382277Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 32523
2022-05-18T03:46:24.2658399Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_bsb3j06
2022-05-18T03:46:24.2659728Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_bsb3j06/_remote_module_non_scriptable.py
2022-05-18T03:46:24.2811735Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7s96_ffj
2022-05-18T03:46:24.2812933Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7s96_ffj/_remote_module_non_scriptable.py
2022-05-18T03:46:24.3329325Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvrboj6ap
2022-05-18T03:46:24.3330140Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn_l0omyv
2022-05-18T03:46:24.3330995Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvrboj6ap/_remote_module_non_scriptable.py
2022-05-18T03:46:24.3331850Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn_l0omyv/_remote_module_non_scriptable.py
2022-05-18T03:46:24.5141280Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:46:24.5312110Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:46:24.5835850Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:46:24.5836448Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:46:24.7418399Z skip: Need at least 1 CUDA device (1.423s)
2022-05-18T03:46:24.7418714Z 
2022-05-18T03:46:24.7419217Z ----------------------------------------------------------------------
2022-05-18T03:46:24.7419510Z Ran 1 test in 1.423s
2022-05-18T03:46:24.7419635Z 
2022-05-18T03:46:24.7419708Z OK (skipped=1)
2022-05-18T03:46:24.7419802Z 
2022-05-18T03:46:24.7419890Z Generating XML reports...
2022-05-18T03:46:24.7451426Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518034623.xml
2022-05-18T03:46:25.4873981Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_9p84vij
2022-05-18T03:46:25.4875041Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_9p84vij/_remote_module_non_scriptable.py
2022-05-18T03:46:25.7339786Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T03:46:25.7349963Z 
2022-05-18T03:46:25.7350548Z Running tests...
2022-05-18T03:46:25.7351185Z ----------------------------------------------------------------------
2022-05-18T03:46:26.0465312Z   test_device_maps_missing_config_remote_response (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32575
2022-05-18T03:46:26.0487928Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32576
2022-05-18T03:46:26.0510627Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 32577
2022-05-18T03:46:26.0536254Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 32578
2022-05-18T03:46:26.6460493Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6ndjvrx2
2022-05-18T03:46:26.6461315Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6ndjvrx2/_remote_module_non_scriptable.py
2022-05-18T03:46:26.6535708Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnd6m5wqb
2022-05-18T03:46:26.6536535Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnd6m5wqb/_remote_module_non_scriptable.py
2022-05-18T03:46:26.6806659Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnmxz4t8d
2022-05-18T03:46:26.6807366Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnmxz4t8d/_remote_module_non_scriptable.py
2022-05-18T03:46:26.6961337Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc7dkfeaz
2022-05-18T03:46:26.6962118Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc7dkfeaz/_remote_module_non_scriptable.py
2022-05-18T03:46:26.8939252Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:46:26.9017265Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:46:26.9307599Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:46:26.9442574Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:46:27.1570789Z skip: Need at least 1 CUDA device (1.422s)
2022-05-18T03:46:27.1571067Z 
2022-05-18T03:46:27.1571624Z ----------------------------------------------------------------------
2022-05-18T03:46:27.1571947Z Ran 1 test in 1.422s
2022-05-18T03:46:27.1572062Z 
2022-05-18T03:46:27.1572121Z OK (skipped=1)
2022-05-18T03:46:27.1572230Z 
2022-05-18T03:46:27.1572314Z Generating XML reports...
2022-05-18T03:46:27.1606691Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518034625.xml
2022-05-18T03:46:27.9213675Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpejxo9rzi
2022-05-18T03:46:27.9214731Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpejxo9rzi/_remote_module_non_scriptable.py
2022-05-18T03:46:28.1715183Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T03:46:28.1724695Z 
2022-05-18T03:46:28.1724824Z Running tests...
2022-05-18T03:46:28.1725362Z ----------------------------------------------------------------------
2022-05-18T03:46:28.4976104Z   test_device_maps_missing_config_response (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32630
2022-05-18T03:46:28.4998472Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32631
2022-05-18T03:46:28.5020930Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 32632
2022-05-18T03:46:28.5045485Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 32633
2022-05-18T03:46:29.0936235Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpol0jja2c
2022-05-18T03:46:29.0937407Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpol0jja2c/_remote_module_non_scriptable.py
2022-05-18T03:46:29.0939583Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa_h7x47d
2022-05-18T03:46:29.0942324Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa_h7x47d/_remote_module_non_scriptable.py
2022-05-18T03:46:29.1408422Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzohdu9jk
2022-05-18T03:46:29.1409569Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq_9a1xqh
2022-05-18T03:46:29.1410634Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzohdu9jk/_remote_module_non_scriptable.py
2022-05-18T03:46:29.1411372Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq_9a1xqh/_remote_module_non_scriptable.py
2022-05-18T03:46:29.3419853Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:46:29.3424561Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:46:29.3911242Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:46:29.4108012Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:46:29.6079939Z skip: Need at least 1 CUDA device (1.435s)
2022-05-18T03:46:29.6080254Z 
2022-05-18T03:46:29.6080652Z ----------------------------------------------------------------------
2022-05-18T03:46:29.6080903Z Ran 1 test in 1.435s
2022-05-18T03:46:29.6081018Z 
2022-05-18T03:46:29.6081097Z OK (skipped=1)
2022-05-18T03:46:29.6081192Z 
2022-05-18T03:46:29.6081280Z Generating XML reports...
2022-05-18T03:46:29.6113404Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518034628.xml
2022-05-18T03:46:30.3573766Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl0lxvoa5
2022-05-18T03:46:30.3574495Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl0lxvoa5/_remote_module_non_scriptable.py
2022-05-18T03:46:30.6046672Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T03:46:30.6056405Z 
2022-05-18T03:46:30.6056902Z Running tests...
2022-05-18T03:46:30.6057331Z ----------------------------------------------------------------------
2022-05-18T03:46:30.9160058Z   test_device_maps_missing_config_response_loop (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32685
2022-05-18T03:46:30.9182005Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32686
2022-05-18T03:46:30.9205364Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 32687
2022-05-18T03:46:30.9228615Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 32688
2022-05-18T03:46:31.4968560Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2qz36w55
2022-05-18T03:46:31.4969702Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2qz36w55/_remote_module_non_scriptable.py
2022-05-18T03:46:31.5428215Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8wgobd7m
2022-05-18T03:46:31.5429117Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8wgobd7m/_remote_module_non_scriptable.py
2022-05-18T03:46:31.5448427Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp59n859_n
2022-05-18T03:46:31.5450186Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp59n859_n/_remote_module_non_scriptable.py
2022-05-18T03:46:31.5475435Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3wq3vifn
2022-05-18T03:46:31.5477779Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3wq3vifn/_remote_module_non_scriptable.py
2022-05-18T03:46:31.7438806Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:46:31.7930150Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:46:31.7934110Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:46:31.7973487Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:46:32.0263292Z skip: Need at least 1 CUDA device (1.420s)
2022-05-18T03:46:32.0263590Z 
2022-05-18T03:46:32.0264112Z ----------------------------------------------------------------------
2022-05-18T03:46:32.0264462Z Ran 1 test in 1.421s
2022-05-18T03:46:32.0264576Z 
2022-05-18T03:46:32.0264635Z OK (skipped=1)
2022-05-18T03:46:32.0264743Z 
2022-05-18T03:46:32.0264830Z Generating XML reports...
2022-05-18T03:46:32.0295910Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518034630.xml
2022-05-18T03:46:32.7720969Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp22s3sp3q
2022-05-18T03:46:32.7721497Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp22s3sp3q/_remote_module_non_scriptable.py
2022-05-18T03:46:33.0197975Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T03:46:33.0207527Z 
2022-05-18T03:46:33.0207680Z Running tests...
2022-05-18T03:46:33.0208291Z ----------------------------------------------------------------------
2022-05-18T03:46:33.3329560Z   test_device_maps_multi_gpu (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32740
2022-05-18T03:46:33.3351594Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32741
2022-05-18T03:46:33.3374144Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 32742
2022-05-18T03:46:33.3398927Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 32743
2022-05-18T03:46:33.9877659Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb6pt9442
2022-05-18T03:46:33.9879098Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb6pt9442/_remote_module_non_scriptable.py
2022-05-18T03:46:34.0376506Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpypc58br2
2022-05-18T03:46:34.0377297Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpypc58br2/_remote_module_non_scriptable.py
2022-05-18T03:46:34.0640535Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf0p2f77b
2022-05-18T03:46:34.0641890Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf0p2f77b/_remote_module_non_scriptable.py
2022-05-18T03:46:34.0899601Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdf84sw9q
2022-05-18T03:46:34.0900379Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdf84sw9q/_remote_module_non_scriptable.py
2022-05-18T03:46:34.2378975Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:46:34.2889081Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:46:34.3135566Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:46:34.3404203Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:46:34.5432998Z skip: Need at least 2 CUDA devices (1.522s)
2022-05-18T03:46:34.5433290Z 
2022-05-18T03:46:34.5433590Z ----------------------------------------------------------------------
2022-05-18T03:46:34.5433914Z Ran 1 test in 1.522s
2022-05-18T03:46:34.5434037Z 
2022-05-18T03:46:34.5434110Z OK (skipped=1)
2022-05-18T03:46:34.5434216Z 
2022-05-18T03:46:34.5434302Z Generating XML reports...
2022-05-18T03:46:34.5465994Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518034633.xml
2022-05-18T03:46:35.2842468Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxif96zas
2022-05-18T03:46:35.2843297Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxif96zas/_remote_module_non_scriptable.py
2022-05-18T03:46:35.5321545Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T03:46:35.5331645Z 
2022-05-18T03:46:35.5331747Z Running tests...
2022-05-18T03:46:35.5332194Z ----------------------------------------------------------------------
2022-05-18T03:46:35.8423310Z   test_device_maps_multi_gpu_self (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 327
2022-05-18T03:46:35.8445393Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 328
2022-05-18T03:46:35.8468378Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 329
2022-05-18T03:46:35.8491449Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 330
2022-05-18T03:46:36.4151554Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpst4k5ti4
2022-05-18T03:46:36.4152286Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpst4k5ti4/_remote_module_non_scriptable.py
2022-05-18T03:46:36.4176759Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2a293ayi
2022-05-18T03:46:36.4179066Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2a293ayi/_remote_module_non_scriptable.py
2022-05-18T03:46:36.4720395Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4isb10wj
2022-05-18T03:46:36.4721170Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4isb10wj/_remote_module_non_scriptable.py
2022-05-18T03:46:36.4721881Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpaksg0myn
2022-05-18T03:46:36.4724426Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpaksg0myn/_remote_module_non_scriptable.py
2022-05-18T03:46:36.6632208Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:46:36.6648907Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:46:36.7212912Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:46:36.7220749Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:46:36.8525679Z skip: Need at least 2 CUDA devices (1.319s)
2022-05-18T03:46:36.8525999Z 
2022-05-18T03:46:36.8526513Z ----------------------------------------------------------------------
2022-05-18T03:46:36.8526948Z Ran 1 test in 1.319s
2022-05-18T03:46:36.8527142Z 
2022-05-18T03:46:36.8527273Z OK (skipped=1)
2022-05-18T03:46:36.8527420Z 
2022-05-18T03:46:36.8527510Z Generating XML reports...
2022-05-18T03:46:36.8559279Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518034635.xml
2022-05-18T03:46:37.6002643Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuva8wsod
2022-05-18T03:46:37.6003315Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuva8wsod/_remote_module_non_scriptable.py
2022-05-18T03:46:37.8488139Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T03:46:37.8497536Z 
2022-05-18T03:46:37.8497664Z Running tests...
2022-05-18T03:46:37.8498117Z ----------------------------------------------------------------------
2022-05-18T03:46:38.1619733Z   test_device_maps_one_to_many (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 382
2022-05-18T03:46:38.1640481Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 383
2022-05-18T03:46:38.1664030Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 384
2022-05-18T03:46:38.1688879Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 385
2022-05-18T03:46:38.8269386Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpigmd9dnm
2022-05-18T03:46:38.8270159Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpigmd9dnm/_remote_module_non_scriptable.py
2022-05-18T03:46:38.8701798Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptxtk_gal
2022-05-18T03:46:38.8702621Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptxtk_gal/_remote_module_non_scriptable.py
2022-05-18T03:46:38.8822566Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy_jq4zr9
2022-05-18T03:46:38.8823783Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy_jq4zr9/_remote_module_non_scriptable.py
2022-05-18T03:46:38.8948161Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps22x_x46
2022-05-18T03:46:38.8949162Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps22x_x46/_remote_module_non_scriptable.py
2022-05-18T03:46:39.0778264Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:46:39.1181963Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:46:39.1312098Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:46:39.1631632Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:46:39.3724393Z skip: Need at least 2 CUDA devices (1.522s)
2022-05-18T03:46:39.3724608Z 
2022-05-18T03:46:39.3725011Z ----------------------------------------------------------------------
2022-05-18T03:46:39.3725266Z Ran 1 test in 1.523s
2022-05-18T03:46:39.3725380Z 
2022-05-18T03:46:39.3725457Z OK (skipped=1)
2022-05-18T03:46:39.3725567Z 
2022-05-18T03:46:39.3725639Z Generating XML reports...
2022-05-18T03:46:39.3759297Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518034637.xml
2022-05-18T03:46:40.1245994Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqstpdscf
2022-05-18T03:46:40.1246665Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqstpdscf/_remote_module_non_scriptable.py
2022-05-18T03:46:40.3726120Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T03:46:40.3735243Z 
2022-05-18T03:46:40.3735325Z Running tests...
2022-05-18T03:46:40.3736355Z ----------------------------------------------------------------------
2022-05-18T03:46:40.6881751Z   test_device_maps_remote (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 437
2022-05-18T03:46:40.6904417Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 438
2022-05-18T03:46:40.6927177Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 439
2022-05-18T03:46:40.6951388Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 440
2022-05-18T03:46:41.2671946Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu35h4cu9
2022-05-18T03:46:41.2672821Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu35h4cu9/_remote_module_non_scriptable.py
2022-05-18T03:46:41.2688422Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptcyososa
2022-05-18T03:46:41.2691356Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptcyososa/_remote_module_non_scriptable.py
2022-05-18T03:46:41.3177864Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplnlxbjg0
2022-05-18T03:46:41.3178561Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1seb5b72
2022-05-18T03:46:41.3179235Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplnlxbjg0/_remote_module_non_scriptable.py
2022-05-18T03:46:41.3180036Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1seb5b72/_remote_module_non_scriptable.py
2022-05-18T03:46:41.5157604Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:46:41.5196094Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:46:41.5689994Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:46:41.5690596Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:46:41.6985283Z skip: Need at least 2 CUDA devices (1.325s)
2022-05-18T03:46:41.6985559Z 
2022-05-18T03:46:41.6985891Z ----------------------------------------------------------------------
2022-05-18T03:46:41.6986194Z Ran 1 test in 1.325s
2022-05-18T03:46:41.6986310Z 
2022-05-18T03:46:41.6986386Z OK (skipped=1)
2022-05-18T03:46:41.6986539Z 
2022-05-18T03:46:41.6986644Z Generating XML reports...
2022-05-18T03:46:41.7017582Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518034640.xml
2022-05-18T03:46:42.4513535Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr3c5msjd
2022-05-18T03:46:42.4514264Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr3c5msjd/_remote_module_non_scriptable.py
2022-05-18T03:46:42.6986417Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T03:46:42.6996037Z 
2022-05-18T03:46:42.6996129Z Running tests...
2022-05-18T03:46:42.6997488Z ----------------------------------------------------------------------
2022-05-18T03:46:43.0138169Z   test_device_maps_return_to_gpu (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 492
2022-05-18T03:46:43.0159898Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 493
2022-05-18T03:46:43.0182321Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 494
2022-05-18T03:46:43.0206737Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 495
2022-05-18T03:46:43.6776887Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpppdsbwcy
2022-05-18T03:46:43.6777660Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpppdsbwcy/_remote_module_non_scriptable.py
2022-05-18T03:46:43.6985886Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1zi28g94
2022-05-18T03:46:43.6986759Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1zi28g94/_remote_module_non_scriptable.py
2022-05-18T03:46:43.7167621Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnraavlh9
2022-05-18T03:46:43.7169585Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnraavlh9/_remote_module_non_scriptable.py
2022-05-18T03:46:43.7286614Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzjdc4o8k
2022-05-18T03:46:43.7288284Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzjdc4o8k/_remote_module_non_scriptable.py
2022-05-18T03:46:43.9271565Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:46:43.9473084Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:46:43.9626889Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:46:43.9774474Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:46:44.1240761Z skip: Need at least 4 CUDA devices (1.424s)
2022-05-18T03:46:44.1241001Z 
2022-05-18T03:46:44.1241358Z ----------------------------------------------------------------------
2022-05-18T03:46:44.1241601Z Ran 1 test in 1.424s
2022-05-18T03:46:44.1241718Z 
2022-05-18T03:46:44.1242061Z OK (skipped=1)
2022-05-18T03:46:44.1242176Z 
2022-05-18T03:46:44.1242264Z Generating XML reports...
2022-05-18T03:46:44.1274381Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518034642.xml
2022-05-18T03:46:44.8715144Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptc5saz5m
2022-05-18T03:46:44.8715914Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptc5saz5m/_remote_module_non_scriptable.py
2022-05-18T03:46:45.1178640Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T03:46:45.1188895Z 
2022-05-18T03:46:45.1189331Z Running tests...
2022-05-18T03:46:45.1189749Z ----------------------------------------------------------------------
2022-05-18T03:46:45.4303075Z   test_device_maps_return_to_gpu_self (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 547
2022-05-18T03:46:45.4325389Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 548
2022-05-18T03:46:45.4348282Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 549
2022-05-18T03:46:45.4371498Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 550
2022-05-18T03:46:46.0607075Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4p0t7r7_
2022-05-18T03:46:46.0608222Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4p0t7r7_/_remote_module_non_scriptable.py
2022-05-18T03:46:46.0736239Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprjazsl8i
2022-05-18T03:46:46.0737018Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprjazsl8i/_remote_module_non_scriptable.py
2022-05-18T03:46:46.0899643Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbo7jfja0
2022-05-18T03:46:46.0900304Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn7i1xlkc
2022-05-18T03:46:46.0900969Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbo7jfja0/_remote_module_non_scriptable.py
2022-05-18T03:46:46.0901655Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn7i1xlkc/_remote_module_non_scriptable.py
2022-05-18T03:46:46.3097371Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:46:46.3208293Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:46:46.3419246Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:46:46.3523921Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:46:46.5406912Z skip: Need at least 4 CUDA devices (1.421s)
2022-05-18T03:46:46.5407241Z 
2022-05-18T03:46:46.5407594Z ----------------------------------------------------------------------
2022-05-18T03:46:46.5407867Z Ran 1 test in 1.422s
2022-05-18T03:46:46.5407983Z 
2022-05-18T03:46:46.5408069Z OK (skipped=1)
2022-05-18T03:46:46.5408209Z 
2022-05-18T03:46:46.5408299Z Generating XML reports...
2022-05-18T03:46:46.5439656Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518034645.xml
2022-05-18T03:46:47.2856624Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptsr475uz
2022-05-18T03:46:47.2857330Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptsr475uz/_remote_module_non_scriptable.py
2022-05-18T03:46:47.5347262Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T03:46:47.5356787Z 
2022-05-18T03:46:47.5356919Z Running tests...
2022-05-18T03:46:47.5357379Z ----------------------------------------------------------------------
2022-05-18T03:46:47.8480732Z   test_device_maps_wrong_worker_name (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 602
2022-05-18T03:46:47.8502615Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 603
2022-05-18T03:46:47.8526220Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 604
2022-05-18T03:46:47.8550063Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 605
2022-05-18T03:46:48.4830048Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphsr48yvm
2022-05-18T03:46:48.4830833Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphsr48yvm/_remote_module_non_scriptable.py
2022-05-18T03:46:48.4934982Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp95x82epv
2022-05-18T03:46:48.4936051Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp95x82epv/_remote_module_non_scriptable.py
2022-05-18T03:46:48.4951829Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjx2de0_1
2022-05-18T03:46:48.4953388Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjx2de0_1/_remote_module_non_scriptable.py
2022-05-18T03:46:48.4998722Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgaua80qh
2022-05-18T03:46:48.5000449Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgaua80qh/_remote_module_non_scriptable.py
2022-05-18T03:46:48.7311858Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:46:48.7411894Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:46:48.7474725Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:46:48.7476779Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:46:48.9584918Z skip: Need at least 2 CUDA devices (1.422s)
2022-05-18T03:46:48.9585193Z 
2022-05-18T03:46:48.9585617Z ----------------------------------------------------------------------
2022-05-18T03:46:48.9586015Z Ran 1 test in 1.423s
2022-05-18T03:46:48.9586203Z 
2022-05-18T03:46:48.9586332Z OK (skipped=1)
2022-05-18T03:46:48.9586505Z 
2022-05-18T03:46:48.9586643Z Generating XML reports...
2022-05-18T03:46:48.9619551Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518034647.xml
2022-05-18T03:46:49.7097892Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdyftv7he
2022-05-18T03:46:49.7098416Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdyftv7he/_remote_module_non_scriptable.py
2022-05-18T03:46:49.9555323Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T03:46:49.9564943Z 
2022-05-18T03:46:49.9565072Z Running tests...
2022-05-18T03:46:49.9565611Z ----------------------------------------------------------------------
2022-05-18T03:46:50.2699661Z   test_device_mismatch (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 657
2022-05-18T03:46:50.2722736Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 658
2022-05-18T03:46:50.2745938Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 659
2022-05-18T03:46:50.2769856Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 660
2022-05-18T03:46:50.8838580Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpave33z_o
2022-05-18T03:46:50.8839334Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpave33z_o/_remote_module_non_scriptable.py
2022-05-18T03:46:50.9157100Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7ug1ody2
2022-05-18T03:46:50.9158635Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7ug1ody2/_remote_module_non_scriptable.py
2022-05-18T03:46:50.9241140Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp482x7rzi
2022-05-18T03:46:50.9242683Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp482x7rzi/_remote_module_non_scriptable.py
2022-05-18T03:46:50.9249914Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf21y563t
2022-05-18T03:46:50.9252033Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf21y563t/_remote_module_non_scriptable.py
2022-05-18T03:46:51.1317472Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:46:51.1728059Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:46:51.1740367Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:46:51.1772268Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:46:51.3805530Z skip: Need at least 1 CUDA device (1.424s)
2022-05-18T03:46:51.3805857Z 
2022-05-18T03:46:51.3806327Z ----------------------------------------------------------------------
2022-05-18T03:46:51.3806713Z Ran 1 test in 1.424s
2022-05-18T03:46:51.3806912Z 
2022-05-18T03:46:51.3807031Z OK (skipped=1)
2022-05-18T03:46:51.3807220Z 
2022-05-18T03:46:51.3807349Z Generating XML reports...
2022-05-18T03:46:51.3840707Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518034649.xml
2022-05-18T03:46:52.1264243Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt4y3j_as
2022-05-18T03:46:52.1265605Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt4y3j_as/_remote_module_non_scriptable.py
2022-05-18T03:46:52.3756828Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T03:46:52.3766690Z 
2022-05-18T03:46:52.3767029Z Running tests...
2022-05-18T03:46:52.3767688Z ----------------------------------------------------------------------
2022-05-18T03:46:52.6890231Z   test_devices_option_mismatch (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 712
2022-05-18T03:46:52.6912160Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 713
2022-05-18T03:46:52.6934876Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 714
2022-05-18T03:46:52.6958665Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 715
2022-05-18T03:46:53.3720866Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp34fshlgy
2022-05-18T03:46:53.3722072Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp34fshlgy/_remote_module_non_scriptable.py
2022-05-18T03:46:53.3952124Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2ztuhony
2022-05-18T03:46:53.3952876Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2ztuhony/_remote_module_non_scriptable.py
2022-05-18T03:46:53.4257412Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp98azx9a2
2022-05-18T03:46:53.4259404Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp98azx9a2/_remote_module_non_scriptable.py
2022-05-18T03:46:53.4591045Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7emirt5c
2022-05-18T03:46:53.4591805Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7emirt5c/_remote_module_non_scriptable.py
2022-05-18T03:46:53.6253669Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:46:53.6469926Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:46:53.6861759Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:46:53.7130571Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:46:53.9094854Z skip: Need at least 2 CUDA devices (1.532s)
2022-05-18T03:46:53.9095151Z 
2022-05-18T03:46:53.9095946Z ----------------------------------------------------------------------
2022-05-18T03:46:53.9096208Z Ran 1 test in 1.533s
2022-05-18T03:46:53.9096324Z 
2022-05-18T03:46:53.9096385Z OK (skipped=1)
2022-05-18T03:46:53.9096495Z 
2022-05-18T03:46:53.9096582Z Generating XML reports...
2022-05-18T03:46:53.9128354Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518034652.xml
2022-05-18T03:46:54.6543853Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwavn_hdw
2022-05-18T03:46:54.6544913Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwavn_hdw/_remote_module_non_scriptable.py
2022-05-18T03:46:54.9018611Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T03:46:54.9028640Z 
2022-05-18T03:46:54.9028727Z Running tests...
2022-05-18T03:46:54.9029205Z ----------------------------------------------------------------------
2022-05-18T03:46:55.2155385Z   test_devices_option_mismatch_reverse (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 767
2022-05-18T03:46:55.2176256Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 768
2022-05-18T03:46:55.2199151Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 769
2022-05-18T03:46:55.2222680Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 770
2022-05-18T03:46:55.8401060Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsetmdns9
2022-05-18T03:46:55.8401877Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsetmdns9/_remote_module_non_scriptable.py
2022-05-18T03:46:55.8640211Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqner7boj
2022-05-18T03:46:55.8641093Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpykubvcho
2022-05-18T03:46:55.8641759Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqner7boj/_remote_module_non_scriptable.py
2022-05-18T03:46:55.8642487Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpykubvcho/_remote_module_non_scriptable.py
2022-05-18T03:46:55.8668206Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkdvgqwfp
2022-05-18T03:46:55.8670246Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkdvgqwfp/_remote_module_non_scriptable.py
2022-05-18T03:46:56.0889037Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:46:56.1132240Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:46:56.1135997Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:46:56.1247194Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:46:56.3258333Z skip: Need at least 2 CUDA devices (1.423s)
2022-05-18T03:46:56.3258692Z 
2022-05-18T03:46:56.3259208Z ----------------------------------------------------------------------
2022-05-18T03:46:56.3259513Z Ran 1 test in 1.423s
2022-05-18T03:46:56.3259627Z 
2022-05-18T03:46:56.3259688Z OK (skipped=1)
2022-05-18T03:46:56.3259797Z 
2022-05-18T03:46:56.3259883Z Generating XML reports...
2022-05-18T03:46:56.3291582Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518034654.xml
2022-05-18T03:46:57.0801599Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe0cly2a3
2022-05-18T03:46:57.0802633Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe0cly2a3/_remote_module_non_scriptable.py
2022-05-18T03:46:57.3287637Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T03:46:57.3297136Z 
2022-05-18T03:46:57.3297567Z Running tests...
2022-05-18T03:46:57.3298416Z ----------------------------------------------------------------------
2022-05-18T03:46:57.6475419Z   test_meta_multiple_tensors (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 822
2022-05-18T03:46:57.6497492Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 823
2022-05-18T03:46:57.6520025Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 824
2022-05-18T03:46:57.6544641Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 825
2022-05-18T03:46:58.3288719Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjarrlu3n
2022-05-18T03:46:58.3289481Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjarrlu3n/_remote_module_non_scriptable.py
2022-05-18T03:46:58.3565709Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9c975qlv
2022-05-18T03:46:58.3566462Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9c975qlv/_remote_module_non_scriptable.py
2022-05-18T03:46:58.3927681Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd1ofuwz0
2022-05-18T03:46:58.3928446Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd1ofuwz0/_remote_module_non_scriptable.py
2022-05-18T03:46:58.3968404Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpquwp4nms
2022-05-18T03:46:58.3969630Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpquwp4nms/_remote_module_non_scriptable.py
2022-05-18T03:46:58.5819795Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:46:58.6079716Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:46:58.6456119Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:46:58.6498892Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:46:58.8580546Z skip: Need at least 1 CUDA device (1.528s)
2022-05-18T03:46:58.8580934Z 
2022-05-18T03:46:58.8581408Z ----------------------------------------------------------------------
2022-05-18T03:46:58.8581662Z Ran 1 test in 1.528s
2022-05-18T03:46:58.8581776Z 
2022-05-18T03:46:58.8581939Z OK (skipped=1)
2022-05-18T03:46:58.8582050Z 
2022-05-18T03:46:58.8582124Z Generating XML reports...
2022-05-18T03:46:58.8615665Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518034657.xml
2022-05-18T03:46:59.6153770Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0xylfaf8
2022-05-18T03:46:59.6154622Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0xylfaf8/_remote_module_non_scriptable.py
2022-05-18T03:46:59.8617486Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T03:46:59.8627326Z 
2022-05-18T03:46:59.8627474Z Running tests...
2022-05-18T03:46:59.8628100Z ----------------------------------------------------------------------
2022-05-18T03:47:00.1749248Z   test_owner_rref_forward_synchronization1 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 877
2022-05-18T03:47:00.1771731Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 878
2022-05-18T03:47:00.1794375Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 879
2022-05-18T03:47:00.1818595Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 880
2022-05-18T03:47:00.8096574Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprfvtw3ya
2022-05-18T03:47:00.8097597Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprfvtw3ya/_remote_module_non_scriptable.py
2022-05-18T03:47:00.8408935Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps93_jodd
2022-05-18T03:47:00.8409739Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps93_jodd/_remote_module_non_scriptable.py
2022-05-18T03:47:00.8419044Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpquvhebmw
2022-05-18T03:47:00.8420596Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpquvhebmw/_remote_module_non_scriptable.py
2022-05-18T03:47:00.8473988Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_x3gcow4
2022-05-18T03:47:00.8475598Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_x3gcow4/_remote_module_non_scriptable.py
2022-05-18T03:47:01.0580395Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:47:01.0906162Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:47:01.0906824Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:47:01.0967212Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:47:01.2852895Z skip: Need at least 1 CUDA device (1.422s)
2022-05-18T03:47:01.2853204Z 
2022-05-18T03:47:01.2853578Z ----------------------------------------------------------------------
2022-05-18T03:47:01.2853832Z Ran 1 test in 1.422s
2022-05-18T03:47:01.2853949Z 
2022-05-18T03:47:01.2854009Z OK (skipped=1)
2022-05-18T03:47:01.2854116Z 
2022-05-18T03:47:01.2854205Z Generating XML reports...
2022-05-18T03:47:01.2885528Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518034659.xml
2022-05-18T03:47:02.0379728Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppwln921t
2022-05-18T03:47:02.0380487Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppwln921t/_remote_module_non_scriptable.py
2022-05-18T03:47:02.2855846Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T03:47:02.2866067Z 
2022-05-18T03:47:02.2866571Z Running tests...
2022-05-18T03:47:02.2866971Z ----------------------------------------------------------------------
2022-05-18T03:47:02.6006647Z   test_owner_rref_forward_synchronization2 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 932
2022-05-18T03:47:02.6028958Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 933
2022-05-18T03:47:02.6051791Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 934
2022-05-18T03:47:02.6075282Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 935
2022-05-18T03:47:03.1806450Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd_4u_1t_
2022-05-18T03:47:03.1807204Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd_4u_1t_/_remote_module_non_scriptable.py
2022-05-18T03:47:03.2200333Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb1py7zes
2022-05-18T03:47:03.2201836Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb1py7zes/_remote_module_non_scriptable.py
2022-05-18T03:47:03.2301026Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu4ynte9v
2022-05-18T03:47:03.2302497Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu4ynte9v/_remote_module_non_scriptable.py
2022-05-18T03:47:03.2366717Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9auep_rw
2022-05-18T03:47:03.2368238Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9auep_rw/_remote_module_non_scriptable.py
2022-05-18T03:47:03.4304814Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:47:03.4710141Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:47:03.4813841Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:47:03.4898094Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:47:03.7112391Z skip: Need at least 2 CUDA devices (1.424s)
2022-05-18T03:47:03.7112580Z 
2022-05-18T03:47:03.7112979Z ----------------------------------------------------------------------
2022-05-18T03:47:03.7113235Z Ran 1 test in 1.425s
2022-05-18T03:47:03.7113353Z 
2022-05-18T03:47:03.7113426Z OK (skipped=1)
2022-05-18T03:47:03.7113533Z 
2022-05-18T03:47:03.7113620Z Generating XML reports...
2022-05-18T03:47:03.7145441Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518034702.xml
2022-05-18T03:47:04.4765376Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplbnr4d0i
2022-05-18T03:47:04.4766184Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplbnr4d0i/_remote_module_non_scriptable.py
2022-05-18T03:47:04.7248358Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T03:47:04.7258267Z 
2022-05-18T03:47:04.7258497Z Running tests...
2022-05-18T03:47:04.7258900Z ----------------------------------------------------------------------
2022-05-18T03:47:05.0404228Z   test_owner_rref_forward_synchronization3 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 987
2022-05-18T03:47:05.0426267Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 988
2022-05-18T03:47:05.0448717Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 989
2022-05-18T03:47:05.0474452Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 990
2022-05-18T03:47:05.6571653Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd2pvdaxb
2022-05-18T03:47:05.6572447Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd2pvdaxb/_remote_module_non_scriptable.py
2022-05-18T03:47:05.7046955Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4fae96_j
2022-05-18T03:47:05.7047775Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4fae96_j/_remote_module_non_scriptable.py
2022-05-18T03:47:05.7229669Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplrvq7zyn
2022-05-18T03:47:05.7230651Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplrvq7zyn/_remote_module_non_scriptable.py
2022-05-18T03:47:05.7334941Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpza0zowoe
2022-05-18T03:47:05.7336563Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpza0zowoe/_remote_module_non_scriptable.py
2022-05-18T03:47:05.9100645Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:47:05.9544569Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:47:05.9720201Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:47:05.9899797Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:47:06.1508838Z skip: Need at least 2 CUDA devices (1.425s)
2022-05-18T03:47:06.1509065Z 
2022-05-18T03:47:06.1509370Z ----------------------------------------------------------------------
2022-05-18T03:47:06.1509611Z Ran 1 test in 1.425s
2022-05-18T03:47:06.1509726Z 
2022-05-18T03:47:06.1509801Z OK (skipped=1)
2022-05-18T03:47:06.1509911Z 
2022-05-18T03:47:06.1509999Z Generating XML reports...
2022-05-18T03:47:06.1541185Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518034704.xml
2022-05-18T03:47:06.8920366Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0bpf5ves
2022-05-18T03:47:06.8921264Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0bpf5ves/_remote_module_non_scriptable.py
2022-05-18T03:47:07.1384520Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T03:47:07.1393635Z 
2022-05-18T03:47:07.1393732Z Running tests...
2022-05-18T03:47:07.1394182Z ----------------------------------------------------------------------
2022-05-18T03:47:07.4554068Z   test_owner_rref_forward_synchronization4 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1042
2022-05-18T03:47:07.4575767Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1043
2022-05-18T03:47:07.4598308Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1044
2022-05-18T03:47:07.4621493Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1045
2022-05-18T03:47:08.1404094Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi88vo2sz
2022-05-18T03:47:08.1405331Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi88vo2sz/_remote_module_non_scriptable.py
2022-05-18T03:47:08.1471040Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp80g2naew
2022-05-18T03:47:08.1472725Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp80g2naew/_remote_module_non_scriptable.py
2022-05-18T03:47:08.1506805Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpynx8_47s
2022-05-18T03:47:08.1508787Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpynx8_47s/_remote_module_non_scriptable.py
2022-05-18T03:47:08.1525069Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn6uo3let
2022-05-18T03:47:08.1526696Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn6uo3let/_remote_module_non_scriptable.py
2022-05-18T03:47:08.3890990Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:47:08.3959065Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:47:08.3994417Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:47:08.4026897Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:47:08.5658566Z skip: Need at least 2 CUDA devices (1.426s)
2022-05-18T03:47:08.5658850Z 
2022-05-18T03:47:08.5659359Z ----------------------------------------------------------------------
2022-05-18T03:47:08.5659710Z Ran 1 test in 1.426s
2022-05-18T03:47:08.5659810Z 
2022-05-18T03:47:08.5659886Z OK (skipped=1)
2022-05-18T03:47:08.5659995Z 
2022-05-18T03:47:08.5660080Z Generating XML reports...
2022-05-18T03:47:08.5693906Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518034707.xml
2022-05-18T03:47:09.3176562Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprholhi_i
2022-05-18T03:47:09.3177031Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprholhi_i/_remote_module_non_scriptable.py
2022-05-18T03:47:09.5676703Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T03:47:09.5686557Z 
2022-05-18T03:47:09.5686680Z Running tests...
2022-05-18T03:47:09.5687257Z ----------------------------------------------------------------------
2022-05-18T03:47:09.8804820Z   test_rref_as_arg_synchronization1 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1097
2022-05-18T03:47:09.8827476Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1098
2022-05-18T03:47:09.8850540Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1099
2022-05-18T03:47:09.8874510Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1100
2022-05-18T03:47:10.5584078Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp91cq4qao
2022-05-18T03:47:10.5584831Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp91cq4qao/_remote_module_non_scriptable.py
2022-05-18T03:47:10.5705167Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppix2fduf
2022-05-18T03:47:10.5706395Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppix2fduf/_remote_module_non_scriptable.py
2022-05-18T03:47:10.5727201Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1da5rrj1
2022-05-18T03:47:10.5728986Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1da5rrj1/_remote_module_non_scriptable.py
2022-05-18T03:47:10.5948435Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3ybczori
2022-05-18T03:47:10.5949725Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3ybczori/_remote_module_non_scriptable.py
2022-05-18T03:47:10.8098282Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:47:10.8190154Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:47:10.8243109Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:47:10.8434389Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:47:10.9910111Z skip: Need at least 1 CUDA device (1.422s)
2022-05-18T03:47:10.9910416Z 
2022-05-18T03:47:10.9910926Z ----------------------------------------------------------------------
2022-05-18T03:47:10.9911204Z Ran 1 test in 1.422s
2022-05-18T03:47:10.9911306Z 
2022-05-18T03:47:10.9911380Z OK (skipped=1)
2022-05-18T03:47:10.9911502Z 
2022-05-18T03:47:10.9911587Z Generating XML reports...
2022-05-18T03:47:10.9942548Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518034709.xml
2022-05-18T03:47:11.7392866Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpupljosc0
2022-05-18T03:47:11.7393308Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpupljosc0/_remote_module_non_scriptable.py
2022-05-18T03:47:11.9883042Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T03:47:11.9892538Z 
2022-05-18T03:47:11.9892649Z Running tests...
2022-05-18T03:47:11.9893233Z ----------------------------------------------------------------------
2022-05-18T03:47:12.3054925Z   test_rref_as_arg_synchronization2 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1152
2022-05-18T03:47:12.3078057Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1153
2022-05-18T03:47:12.3101382Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1154
2022-05-18T03:47:12.3125433Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1155
2022-05-18T03:47:12.9379380Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvy_8d87d
2022-05-18T03:47:12.9380163Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvy_8d87d/_remote_module_non_scriptable.py
2022-05-18T03:47:12.9715079Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1dx4b6qp
2022-05-18T03:47:12.9715866Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1dx4b6qp/_remote_module_non_scriptable.py
2022-05-18T03:47:12.9832846Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprcljy8_1
2022-05-18T03:47:12.9834103Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprcljy8_1/_remote_module_non_scriptable.py
2022-05-18T03:47:12.9892980Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk56ywkpa
2022-05-18T03:47:12.9894803Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk56ywkpa/_remote_module_non_scriptable.py
2022-05-18T03:47:13.1844439Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:47:13.2204159Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:47:13.2332943Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:47:13.2487954Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:47:13.4159010Z skip: Need at least 2 CUDA devices (1.426s)
2022-05-18T03:47:13.4159264Z 
2022-05-18T03:47:13.4159757Z ----------------------------------------------------------------------
2022-05-18T03:47:13.4160205Z Ran 1 test in 1.427s
2022-05-18T03:47:13.4160427Z 
2022-05-18T03:47:13.4160550Z OK (skipped=1)
2022-05-18T03:47:13.4160743Z 
2022-05-18T03:47:13.4160900Z Generating XML reports...
2022-05-18T03:47:13.4192559Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518034711.xml
2022-05-18T03:47:14.1661361Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbaqnl20l
2022-05-18T03:47:14.1662143Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbaqnl20l/_remote_module_non_scriptable.py
2022-05-18T03:47:14.4121543Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T03:47:14.4131156Z 
2022-05-18T03:47:14.4131298Z Running tests...
2022-05-18T03:47:14.4131784Z ----------------------------------------------------------------------
2022-05-18T03:47:14.7279728Z   test_rref_as_arg_synchronization3 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1207
2022-05-18T03:47:14.7301543Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1208
2022-05-18T03:47:14.7325053Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1209
2022-05-18T03:47:14.7348739Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1210
2022-05-18T03:47:15.3006446Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp64ne1rz7
2022-05-18T03:47:15.3007228Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp64ne1rz7/_remote_module_non_scriptable.py
2022-05-18T03:47:15.3083542Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3bsz9wtk
2022-05-18T03:47:15.3084792Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3bsz9wtk/_remote_module_non_scriptable.py
2022-05-18T03:47:15.3161600Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1evx5e0c
2022-05-18T03:47:15.3162735Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1evx5e0c/_remote_module_non_scriptable.py
2022-05-18T03:47:15.3599545Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprmx9_ect
2022-05-18T03:47:15.3600613Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprmx9_ect/_remote_module_non_scriptable.py
2022-05-18T03:47:15.5528617Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:47:15.5570503Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:47:15.5654472Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:47:15.6098963Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:47:15.7381740Z skip: Need at least 2 CUDA devices (1.325s)
2022-05-18T03:47:15.7381989Z 
2022-05-18T03:47:15.7382376Z ----------------------------------------------------------------------
2022-05-18T03:47:15.7383073Z Ran 1 test in 1.325s
2022-05-18T03:47:15.7383192Z 
2022-05-18T03:47:15.7383274Z OK (skipped=1)
2022-05-18T03:47:15.7383383Z 
2022-05-18T03:47:15.7383535Z Generating XML reports...
2022-05-18T03:47:15.7415660Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518034714.xml
2022-05-18T03:47:16.4824896Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi86y3ddc
2022-05-18T03:47:16.4825791Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi86y3ddc/_remote_module_non_scriptable.py
2022-05-18T03:47:16.7303844Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T03:47:16.7313879Z 
2022-05-18T03:47:16.7314006Z Running tests...
2022-05-18T03:47:16.7314589Z ----------------------------------------------------------------------
2022-05-18T03:47:17.0432399Z   test_rref_as_arg_synchronization4 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1262
2022-05-18T03:47:17.0455397Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1263
2022-05-18T03:47:17.0478236Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1264
2022-05-18T03:47:17.0502180Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1265
2022-05-18T03:47:17.6536801Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4fbyxq0d
2022-05-18T03:47:17.6538074Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4fbyxq0d/_remote_module_non_scriptable.py
2022-05-18T03:47:17.6695717Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp02h5lr_6
2022-05-18T03:47:17.6696503Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp02h5lr_6/_remote_module_non_scriptable.py
2022-05-18T03:47:17.6959567Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0lf60y7n
2022-05-18T03:47:17.6961074Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0lf60y7n/_remote_module_non_scriptable.py
2022-05-18T03:47:17.7049499Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp25d8fd06
2022-05-18T03:47:17.7050593Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp25d8fd06/_remote_module_non_scriptable.py
2022-05-18T03:47:17.9020324Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:47:17.9182799Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:47:17.9431103Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:47:17.9559957Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:47:18.1536592Z skip: Need at least 2 CUDA devices (1.422s)
2022-05-18T03:47:18.1536808Z 
2022-05-18T03:47:18.1544017Z ----------------------------------------------------------------------
2022-05-18T03:47:18.1544315Z Ran 1 test in 1.422s
2022-05-18T03:47:18.1544445Z 
2022-05-18T03:47:18.1544522Z OK (skipped=1)
2022-05-18T03:47:18.1544630Z 
2022-05-18T03:47:18.1544702Z Generating XML reports...
2022-05-18T03:47:18.1569792Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518034716.xml
2022-05-18T03:47:18.8926773Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr5gir6iy
2022-05-18T03:47:18.8927531Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr5gir6iy/_remote_module_non_scriptable.py
2022-05-18T03:47:19.1405328Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T03:47:19.1415608Z 
2022-05-18T03:47:19.1415929Z Running tests...
2022-05-18T03:47:19.1416863Z ----------------------------------------------------------------------
2022-05-18T03:47:19.4538646Z   test_rref_as_arg_synchronization5 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1317
2022-05-18T03:47:19.4561073Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1318
2022-05-18T03:47:19.4583703Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1319
2022-05-18T03:47:19.4607028Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1320
2022-05-18T03:47:20.0822357Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpntkr48lg
2022-05-18T03:47:20.0823305Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpntkr48lg/_remote_module_non_scriptable.py
2022-05-18T03:47:20.0947668Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl_891ddx
2022-05-18T03:47:20.0948692Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl_891ddx/_remote_module_non_scriptable.py
2022-05-18T03:47:20.1024606Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6b3qn2ym
2022-05-18T03:47:20.1026499Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6b3qn2ym/_remote_module_non_scriptable.py
2022-05-18T03:47:20.1166222Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy8i2wm2n
2022-05-18T03:47:20.1167811Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy8i2wm2n/_remote_module_non_scriptable.py
2022-05-18T03:47:20.3285001Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:47:20.3421649Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:47:20.3524177Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:47:20.3648912Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:47:20.5641306Z skip: Need at least 1 CUDA device (1.422s)
2022-05-18T03:47:20.5641583Z 
2022-05-18T03:47:20.5642041Z ----------------------------------------------------------------------
2022-05-18T03:47:20.5642455Z Ran 1 test in 1.422s
2022-05-18T03:47:20.5642626Z 
2022-05-18T03:47:20.5642721Z OK (skipped=1)
2022-05-18T03:47:20.5642885Z 
2022-05-18T03:47:20.5643027Z Generating XML reports...
2022-05-18T03:47:20.5675588Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518034719.xml
2022-05-18T03:47:21.3078071Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplvpkx__y
2022-05-18T03:47:21.3078730Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplvpkx__y/_remote_module_non_scriptable.py
2022-05-18T03:47:21.5538350Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T03:47:21.5548638Z 
2022-05-18T03:47:21.5548759Z Running tests...
2022-05-18T03:47:21.5549472Z ----------------------------------------------------------------------
2022-05-18T03:47:21.8647807Z   test_rref_forward_synchronization1 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1372
2022-05-18T03:47:21.8670433Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1373
2022-05-18T03:47:21.8693080Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1374
2022-05-18T03:47:21.8717030Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1375
2022-05-18T03:47:22.5677245Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmo3vq4hu
2022-05-18T03:47:22.5678042Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmo3vq4hu/_remote_module_non_scriptable.py
2022-05-18T03:47:22.6042869Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj8zvke1e
2022-05-18T03:47:22.6044052Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj8zvke1e/_remote_module_non_scriptable.py
2022-05-18T03:47:22.6075922Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn0ud_are
2022-05-18T03:47:22.6077890Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn0ud_are/_remote_module_non_scriptable.py
2022-05-18T03:47:22.6319923Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3j0_y18a
2022-05-18T03:47:22.6320916Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3j0_y18a/_remote_module_non_scriptable.py
2022-05-18T03:47:22.8166511Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:47:22.8561481Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:47:22.8609310Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:47:22.8799889Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:47:23.0752990Z skip: Need at least 1 CUDA device (1.520s)
2022-05-18T03:47:23.0753305Z 
2022-05-18T03:47:23.0753770Z ----------------------------------------------------------------------
2022-05-18T03:47:23.0754154Z Ran 1 test in 1.520s
2022-05-18T03:47:23.0754331Z 
2022-05-18T03:47:23.0754443Z OK (skipped=1)
2022-05-18T03:47:23.0754601Z 
2022-05-18T03:47:23.0754720Z Generating XML reports...
2022-05-18T03:47:23.0787028Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518034721.xml
2022-05-18T03:47:23.8143080Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppjwnk9s2
2022-05-18T03:47:23.8144255Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppjwnk9s2/_remote_module_non_scriptable.py
2022-05-18T03:47:24.0615320Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T03:47:24.0625317Z 
2022-05-18T03:47:24.0625456Z Running tests...
2022-05-18T03:47:24.0626465Z ----------------------------------------------------------------------
2022-05-18T03:47:24.3749376Z   test_rref_forward_synchronization2 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1427
2022-05-18T03:47:24.3771735Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1428
2022-05-18T03:47:24.3796057Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1429
2022-05-18T03:47:24.3819804Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1430
2022-05-18T03:47:25.0277128Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptnatatxx
2022-05-18T03:47:25.0277940Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptnatatxx/_remote_module_non_scriptable.py
2022-05-18T03:47:25.0511464Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpain_trxf
2022-05-18T03:47:25.0512178Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi7kyz82w
2022-05-18T03:47:25.0512857Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpain_trxf/_remote_module_non_scriptable.py
2022-05-18T03:47:25.0515333Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi7kyz82w/_remote_module_non_scriptable.py
2022-05-18T03:47:25.0516927Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzsrnpnid
2022-05-18T03:47:25.0520237Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzsrnpnid/_remote_module_non_scriptable.py
2022-05-18T03:47:25.2808046Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:47:25.3008975Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:47:25.3022673Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:47:25.3105845Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:47:25.4853872Z skip: Need at least 2 CUDA devices (1.423s)
2022-05-18T03:47:25.4854221Z 
2022-05-18T03:47:25.4854677Z ----------------------------------------------------------------------
2022-05-18T03:47:25.4855013Z Ran 1 test in 1.423s
2022-05-18T03:47:25.4855128Z 
2022-05-18T03:47:25.4855203Z OK (skipped=1)
2022-05-18T03:47:25.4855314Z 
2022-05-18T03:47:25.4855400Z Generating XML reports...
2022-05-18T03:47:25.4887482Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518034724.xml
2022-05-18T03:47:26.2290119Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7udqoybv
2022-05-18T03:47:26.2290611Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7udqoybv/_remote_module_non_scriptable.py
2022-05-18T03:47:26.4749197Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T03:47:26.4759153Z 
2022-05-18T03:47:26.4759290Z Running tests...
2022-05-18T03:47:26.4759857Z ----------------------------------------------------------------------
2022-05-18T03:47:26.7865043Z   test_rref_forward_synchronization3 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1482
2022-05-18T03:47:26.7886852Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1483
2022-05-18T03:47:26.7909222Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1484
2022-05-18T03:47:26.7932792Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1485
2022-05-18T03:47:27.4255443Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzpm19zn2
2022-05-18T03:47:27.4256248Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzpm19zn2/_remote_module_non_scriptable.py
2022-05-18T03:47:27.4269393Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwa95sfiy
2022-05-18T03:47:27.4270613Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwa95sfiy/_remote_module_non_scriptable.py
2022-05-18T03:47:27.4286379Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6g1n9zq2
2022-05-18T03:47:27.4287646Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6g1n9zq2/_remote_module_non_scriptable.py
2022-05-18T03:47:27.4358444Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp21rwc2z7
2022-05-18T03:47:27.4360141Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp21rwc2z7/_remote_module_non_scriptable.py
2022-05-18T03:47:27.6751559Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:47:27.6767353Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:47:27.6779196Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:47:27.6829059Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:47:27.8968461Z skip: Need at least 2 CUDA devices (1.421s)
2022-05-18T03:47:27.8968783Z 
2022-05-18T03:47:27.8969283Z ----------------------------------------------------------------------
2022-05-18T03:47:27.8969560Z Ran 1 test in 1.421s
2022-05-18T03:47:27.8969676Z 
2022-05-18T03:47:27.8969751Z OK (skipped=1)
2022-05-18T03:47:27.8969859Z 
2022-05-18T03:47:27.8969931Z Generating XML reports...
2022-05-18T03:47:27.9001617Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518034726.xml
2022-05-18T03:47:28.6471634Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzt8jbfc8
2022-05-18T03:47:28.6472362Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzt8jbfc8/_remote_module_non_scriptable.py
2022-05-18T03:47:28.8956328Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T03:47:28.8965604Z 
2022-05-18T03:47:28.8965698Z Running tests...
2022-05-18T03:47:28.8966362Z ----------------------------------------------------------------------
2022-05-18T03:47:29.2099593Z   test_rref_forward_synchronization4 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1537
2022-05-18T03:47:29.2122063Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1538
2022-05-18T03:47:29.2145053Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1539
2022-05-18T03:47:29.2168830Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1540
2022-05-18T03:47:29.8814415Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8gj1n9vg
2022-05-18T03:47:29.8815194Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8gj1n9vg/_remote_module_non_scriptable.py
2022-05-18T03:47:29.8929482Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgdi8kf5f
2022-05-18T03:47:29.8930307Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgdi8kf5f/_remote_module_non_scriptable.py
2022-05-18T03:47:29.8965003Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8divhj1s
2022-05-18T03:47:29.8967224Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8divhj1s/_remote_module_non_scriptable.py
2022-05-18T03:47:29.9228261Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz6u9r39s
2022-05-18T03:47:29.9229040Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz6u9r39s/_remote_module_non_scriptable.py
2022-05-18T03:47:30.1308385Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:47:30.1459226Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:47:30.1471044Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:47:30.1829414Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:47:30.3204723Z skip: Need at least 2 CUDA devices (1.424s)
2022-05-18T03:47:30.3205047Z 
2022-05-18T03:47:30.3205528Z ----------------------------------------------------------------------
2022-05-18T03:47:30.3205780Z Ran 1 test in 1.424s
2022-05-18T03:47:30.3205895Z 
2022-05-18T03:47:30.3205955Z OK (skipped=1)
2022-05-18T03:47:30.3206065Z 
2022-05-18T03:47:30.3206153Z Generating XML reports...
2022-05-18T03:47:30.3237731Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518034728.xml
2022-05-18T03:47:31.0766500Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx9suo_bp
2022-05-18T03:47:31.0767324Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx9suo_bp/_remote_module_non_scriptable.py
2022-05-18T03:47:31.3232495Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T03:47:31.3242298Z 
2022-05-18T03:47:31.3242507Z Running tests...
2022-05-18T03:47:31.3243033Z ----------------------------------------------------------------------
2022-05-18T03:47:31.6360815Z   test_rref_to_here_synchronization1 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1592
2022-05-18T03:47:31.6382111Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1593
2022-05-18T03:47:31.6404984Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1594
2022-05-18T03:47:31.6429304Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1595
2022-05-18T03:47:32.2733223Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp516dlo6r
2022-05-18T03:47:32.2735903Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp516dlo6r/_remote_module_non_scriptable.py
2022-05-18T03:47:32.3249923Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbhc_g6j2
2022-05-18T03:47:32.3250669Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbhc_g6j2/_remote_module_non_scriptable.py
2022-05-18T03:47:32.3269245Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3_4s2h_4
2022-05-18T03:47:32.3270795Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3_4s2h_4/_remote_module_non_scriptable.py
2022-05-18T03:47:32.3299937Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjaf9gbbt
2022-05-18T03:47:32.3301652Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjaf9gbbt/_remote_module_non_scriptable.py
2022-05-18T03:47:32.5237818Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:47:32.5763565Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:47:32.5764037Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:47:32.5894997Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:47:32.7463651Z skip: Need at least 1 CUDA device (1.422s)
2022-05-18T03:47:32.7463898Z 
2022-05-18T03:47:32.7464219Z ----------------------------------------------------------------------
2022-05-18T03:47:32.7464472Z Ran 1 test in 1.422s
2022-05-18T03:47:32.7464594Z 
2022-05-18T03:47:32.7464712Z OK (skipped=1)
2022-05-18T03:47:32.7464825Z 
2022-05-18T03:47:32.7464913Z Generating XML reports...
2022-05-18T03:47:32.7496868Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518034731.xml
2022-05-18T03:47:33.5000409Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzv9bcda6
2022-05-18T03:47:33.5001115Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzv9bcda6/_remote_module_non_scriptable.py
2022-05-18T03:47:33.7507883Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T03:47:33.7517745Z 
2022-05-18T03:47:33.7517853Z Running tests...
2022-05-18T03:47:33.7518417Z ----------------------------------------------------------------------
2022-05-18T03:47:34.0665114Z   test_rref_to_here_synchronization2 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1647
2022-05-18T03:47:34.0686666Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1648
2022-05-18T03:47:34.0709720Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1649
2022-05-18T03:47:34.0734375Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1650
2022-05-18T03:47:34.7405305Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyzuqguto
2022-05-18T03:47:34.7406046Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyzuqguto/_remote_module_non_scriptable.py
2022-05-18T03:47:34.7942662Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppkt00p_t
2022-05-18T03:47:34.7943560Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppkt00p_t/_remote_module_non_scriptable.py
2022-05-18T03:47:34.8095860Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn_96_m4i
2022-05-18T03:47:34.8096618Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn_96_m4i/_remote_module_non_scriptable.py
2022-05-18T03:47:34.8466054Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjht8lrd2
2022-05-18T03:47:34.8467520Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjht8lrd2/_remote_module_non_scriptable.py
2022-05-18T03:47:34.9900221Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:47:35.0444962Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:47:35.0897968Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:47:35.1224353Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:47:35.2770343Z skip: Need at least 2 CUDA devices (1.525s)
2022-05-18T03:47:35.2770583Z 
2022-05-18T03:47:35.2770990Z ----------------------------------------------------------------------
2022-05-18T03:47:35.2771246Z Ran 1 test in 1.525s
2022-05-18T03:47:35.2771361Z 
2022-05-18T03:47:35.2771440Z OK (skipped=1)
2022-05-18T03:47:35.2771550Z 
2022-05-18T03:47:35.2771622Z Generating XML reports...
2022-05-18T03:47:35.2805904Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518034733.xml
2022-05-18T03:47:36.0183049Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm64fdnue
2022-05-18T03:47:36.0183884Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm64fdnue/_remote_module_non_scriptable.py
2022-05-18T03:47:36.2648661Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T03:47:36.2658073Z 
2022-05-18T03:47:36.2658359Z Running tests...
2022-05-18T03:47:36.2659023Z ----------------------------------------------------------------------
2022-05-18T03:47:36.5770958Z   test_rref_to_here_synchronization3 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1702
2022-05-18T03:47:36.5794167Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1703
2022-05-18T03:47:36.5818208Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1704
2022-05-18T03:47:36.5841648Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1705
2022-05-18T03:47:37.2272425Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf9wqxgq3
2022-05-18T03:47:37.2273204Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf9wqxgq3/_remote_module_non_scriptable.py
2022-05-18T03:47:37.2392601Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphsjt7f79
2022-05-18T03:47:37.2393756Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphsjt7f79/_remote_module_non_scriptable.py
2022-05-18T03:47:37.2686894Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5aqktz3p
2022-05-18T03:47:37.2687661Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5aqktz3p/_remote_module_non_scriptable.py
2022-05-18T03:47:37.3160676Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2894ia7o
2022-05-18T03:47:37.3161899Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2894ia7o/_remote_module_non_scriptable.py
2022-05-18T03:47:37.4817293Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:47:37.4902597Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:47:37.5339217Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:47:37.5723676Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:47:37.7878474Z skip: Need at least 2 CUDA devices (1.522s)
2022-05-18T03:47:37.7878804Z 
2022-05-18T03:47:37.7879320Z ----------------------------------------------------------------------
2022-05-18T03:47:37.7879580Z Ran 1 test in 1.522s
2022-05-18T03:47:37.7879696Z 
2022-05-18T03:47:37.7879771Z OK (skipped=1)
2022-05-18T03:47:37.7880123Z 
2022-05-18T03:47:37.7880196Z Generating XML reports...
2022-05-18T03:47:37.7911797Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518034736.xml
2022-05-18T03:47:38.5471972Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2ydovaba
2022-05-18T03:47:38.5472907Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2ydovaba/_remote_module_non_scriptable.py
2022-05-18T03:47:38.7969060Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T03:47:38.7978466Z 
2022-05-18T03:47:38.7978557Z Running tests...
2022-05-18T03:47:38.7979107Z ----------------------------------------------------------------------
2022-05-18T03:47:39.1178220Z   test_rref_to_here_synchronization4 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1757
2022-05-18T03:47:39.1200866Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1758
2022-05-18T03:47:39.1223470Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1759
2022-05-18T03:47:39.1248339Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1760
2022-05-18T03:47:39.7833487Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnea2ber2
2022-05-18T03:47:39.7834291Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnea2ber2/_remote_module_non_scriptable.py
2022-05-18T03:47:39.8091561Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo0p8xgvg
2022-05-18T03:47:39.8092541Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo0p8xgvg/_remote_module_non_scriptable.py
2022-05-18T03:47:39.8368059Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq3jw3q8m
2022-05-18T03:47:39.8368870Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq3jw3q8m/_remote_module_non_scriptable.py
2022-05-18T03:47:39.8679978Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn4_cdz20
2022-05-18T03:47:39.8680957Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn4_cdz20/_remote_module_non_scriptable.py
2022-05-18T03:47:40.0340410Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:47:40.0601443Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:47:40.0864516Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:47:40.1520026Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:47:40.3284951Z skip: Need at least 2 CUDA devices (1.530s)
2022-05-18T03:47:40.3285202Z 
2022-05-18T03:47:40.3285729Z ----------------------------------------------------------------------
2022-05-18T03:47:40.3286069Z Ran 1 test in 1.531s
2022-05-18T03:47:40.3286202Z 
2022-05-18T03:47:40.3286276Z OK (skipped=1)
2022-05-18T03:47:40.3286385Z 
2022-05-18T03:47:40.3286460Z Generating XML reports...
2022-05-18T03:47:40.3318357Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518034738.xml
2022-05-18T03:47:41.0855714Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpozf2o_rs
2022-05-18T03:47:41.0856456Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpozf2o_rs/_remote_module_non_scriptable.py
2022-05-18T03:47:41.3380395Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T03:47:41.3390962Z 
2022-05-18T03:47:41.6631675Z Running tests...
2022-05-18T03:47:41.6632382Z ----------------------------------------------------------------------
2022-05-18T03:47:41.6633202Z   test_rref_with_unpickleable_attributes (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1812
2022-05-18T03:47:41.6654926Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1813
2022-05-18T03:47:41.6677491Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1814
2022-05-18T03:47:41.6700638Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1815
2022-05-18T03:47:42.2916835Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp587l6gg_
2022-05-18T03:47:42.2917839Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp587l6gg_/_remote_module_non_scriptable.py
2022-05-18T03:47:42.3116601Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppfup53cf
2022-05-18T03:47:42.3117378Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppfup53cf/_remote_module_non_scriptable.py
2022-05-18T03:47:42.3254724Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwo8jygpj
2022-05-18T03:47:42.3255475Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwo8jygpj/_remote_module_non_scriptable.py
2022-05-18T03:47:42.3435734Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg7vf1vbo
2022-05-18T03:47:42.3436698Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg7vf1vbo/_remote_module_non_scriptable.py
2022-05-18T03:47:42.5423844Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:47:42.5596425Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:47:42.5789909Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:47:42.5926762Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:47:42.7735481Z skip: Need at least 1 CUDA device (1.434s)
2022-05-18T03:47:42.7735690Z 
2022-05-18T03:47:42.7735991Z ----------------------------------------------------------------------
2022-05-18T03:47:42.7736327Z Ran 1 test in 1.434s
2022-05-18T03:47:42.7736429Z 
2022-05-18T03:47:42.7736506Z OK (skipped=1)
2022-05-18T03:47:42.7736615Z 
2022-05-18T03:47:42.7736708Z Generating XML reports...
2022-05-18T03:47:42.7771022Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518034741.xml
2022-05-18T03:47:43.5173303Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqhucihd0
2022-05-18T03:47:43.5173954Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqhucihd0/_remote_module_non_scriptable.py
2022-05-18T03:47:43.7635513Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T03:47:43.7645906Z 
2022-05-18T03:47:43.7646164Z Running tests...
2022-05-18T03:47:43.7646822Z ----------------------------------------------------------------------
2022-05-18T03:47:44.0817797Z   test_tensor_view_as_return_value (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1867
2022-05-18T03:47:44.0840083Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1868
2022-05-18T03:47:44.0862824Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1869
2022-05-18T03:47:44.0886232Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1870
2022-05-18T03:47:44.6961043Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4ggsz25h
2022-05-18T03:47:44.6961986Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4ggsz25h/_remote_module_non_scriptable.py
2022-05-18T03:47:44.7395708Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp58eo5d5b
2022-05-18T03:47:44.7396465Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi6_ldy6h
2022-05-18T03:47:44.7397396Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp58eo5d5b/_remote_module_non_scriptable.py
2022-05-18T03:47:44.7400475Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi6_ldy6h/_remote_module_non_scriptable.py
2022-05-18T03:47:44.7610875Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcti46im8
2022-05-18T03:47:44.7611845Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcti46im8/_remote_module_non_scriptable.py
2022-05-18T03:47:44.9461613Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:47:44.9870815Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:47:45.0048745Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:47:45.0096562Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:47:45.1920651Z skip: Need at least 1 CUDA device (1.427s)
2022-05-18T03:47:45.1920960Z 
2022-05-18T03:47:45.1921472Z ----------------------------------------------------------------------
2022-05-18T03:47:45.1921889Z Ran 1 test in 1.427s
2022-05-18T03:47:45.1922005Z 
2022-05-18T03:47:45.1922086Z OK (skipped=1)
2022-05-18T03:47:45.1922195Z 
2022-05-18T03:47:45.1922282Z Generating XML reports...
2022-05-18T03:47:45.1953937Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518034743.xml
2022-05-18T03:47:45.9359058Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpejxunerf
2022-05-18T03:47:45.9359905Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpejxunerf/_remote_module_non_scriptable.py
2022-05-18T03:47:46.1818593Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T03:47:46.1828220Z 
2022-05-18T03:47:46.1828368Z Running tests...
2022-05-18T03:47:46.1828762Z ----------------------------------------------------------------------
2022-05-18T03:47:46.4973908Z   test_device_maps_backward_pass (__main__.TensorPipeTensorPipeCudaDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1922
2022-05-18T03:47:46.4996138Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1923
2022-05-18T03:47:46.5019873Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1924
2022-05-18T03:47:46.5044050Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1925
2022-05-18T03:47:47.1501229Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd8146whi
2022-05-18T03:47:47.1502073Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd8146whi/_remote_module_non_scriptable.py
2022-05-18T03:47:47.1522879Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpln4wjfyx
2022-05-18T03:47:47.1525360Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpln4wjfyx/_remote_module_non_scriptable.py
2022-05-18T03:47:47.2323230Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp_h98pum
2022-05-18T03:47:47.2324615Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp_h98pum/_remote_module_non_scriptable.py
2022-05-18T03:47:47.2325380Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5gl9kty2
2022-05-18T03:47:47.2327409Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5gl9kty2/_remote_module_non_scriptable.py
2022-05-18T03:47:47.4005677Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:47:47.4006341Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:47:47.4826120Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:47:47.5067301Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:47:47.7079592Z skip: Need at least 4 CUDA devices (1.525s)
2022-05-18T03:47:47.7079879Z 
2022-05-18T03:47:47.7080507Z ----------------------------------------------------------------------
2022-05-18T03:47:47.7080766Z Ran 1 test in 1.525s
2022-05-18T03:47:47.7080881Z 
2022-05-18T03:47:47.7080955Z OK (skipped=1)
2022-05-18T03:47:47.7081061Z 
2022-05-18T03:47:47.7081134Z Generating XML reports...
2022-05-18T03:47:47.7113530Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeCudaDistAutogradTest-20220518034746.xml
2022-05-18T03:47:48.4493879Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp33km7b7_
2022-05-18T03:47:48.4494573Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp33km7b7_/_remote_module_non_scriptable.py
2022-05-18T03:47:48.6952271Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T03:47:48.6962048Z 
2022-05-18T03:47:48.6962346Z Running tests...
2022-05-18T03:47:48.6963036Z ----------------------------------------------------------------------
2022-05-18T03:47:49.0063165Z   test_dist_autograd_sync_streams (__main__.TensorPipeTensorPipeCudaDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1977
2022-05-18T03:47:49.0084787Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1978
2022-05-18T03:47:49.0107604Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1979
2022-05-18T03:47:49.0131435Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1980
2022-05-18T03:47:49.6594817Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcwxb217s
2022-05-18T03:47:49.6596367Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcwxb217s/_remote_module_non_scriptable.py
2022-05-18T03:47:49.6727409Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqqdegu64
2022-05-18T03:47:49.6728561Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqqdegu64/_remote_module_non_scriptable.py
2022-05-18T03:47:49.6835177Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp24ho515e
2022-05-18T03:47:49.6836525Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp24ho515e/_remote_module_non_scriptable.py
2022-05-18T03:47:49.6947735Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjjav83n7
2022-05-18T03:47:49.6949538Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjjav83n7/_remote_module_non_scriptable.py
2022-05-18T03:47:49.9073115Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:47:49.9225259Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:47:49.9315579Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:47:49.9404972Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:47:50.1166249Z skip: Need at least 4 CUDA devices (1.420s)
2022-05-18T03:47:50.1166556Z 
2022-05-18T03:47:50.1167076Z ----------------------------------------------------------------------
2022-05-18T03:47:50.1167447Z Ran 1 test in 1.420s
2022-05-18T03:47:50.1167563Z 
2022-05-18T03:47:50.1167635Z OK (skipped=1)
2022-05-18T03:47:50.1167742Z 
2022-05-18T03:47:50.1167819Z Generating XML reports...
2022-05-18T03:47:50.1199000Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeCudaDistAutogradTest-20220518034748.xml
2022-05-18T03:47:50.8572081Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyg4n2r0b
2022-05-18T03:47:50.8572615Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyg4n2r0b/_remote_module_non_scriptable.py
2022-05-18T03:47:51.1036063Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-05-18T03:47:51.1044978Z 
2022-05-18T03:47:51.1045077Z Running tests...
2022-05-18T03:47:51.1045568Z ----------------------------------------------------------------------
2022-05-18T03:47:51.4168503Z   test_gradients_synchronizations (__main__.TensorPipeTensorPipeCudaDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2032
2022-05-18T03:47:51.4190964Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2033
2022-05-18T03:47:51.4214273Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2034
2022-05-18T03:47:51.4238365Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2035
2022-05-18T03:47:52.0542644Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp89i5crvz
2022-05-18T03:47:52.0543591Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp89i5crvz/_remote_module_non_scriptable.py
2022-05-18T03:47:52.0772755Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp188dqo10
2022-05-18T03:47:52.0774439Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp188dqo10/_remote_module_non_scriptable.py
2022-05-18T03:47:52.0867450Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1sct40o2
2022-05-18T03:47:52.0869376Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1sct40o2/_remote_module_non_scriptable.py
2022-05-18T03:47:52.1053627Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpim4db_0e
2022-05-18T03:47:52.1055231Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpim4db_0e/_remote_module_non_scriptable.py
2022-05-18T03:47:52.3044766Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:47:52.3275743Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:47:52.3482213Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:47:52.3660892Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:47:52.5272970Z skip: Need at least 4 CUDA devices (1.423s)
2022-05-18T03:47:52.5273367Z 
2022-05-18T03:47:52.5274014Z ----------------------------------------------------------------------
2022-05-18T03:47:52.5274416Z Ran 1 test in 1.423s
2022-05-18T03:47:52.5274613Z 
2022-05-18T03:47:52.5274726Z OK (skipped=1)
2022-05-18T03:47:52.5274908Z 
2022-05-18T03:47:52.5275048Z Generating XML reports...
2022-05-18T03:47:52.5307572Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeCudaDistAutogradTest-20220518034751.xml
2022-05-18T03:47:52.8424193Z Running distributed/rpc/test_faulty_agent ... [2022-05-18 03:47:52.841954]
2022-05-18T03:47:52.8424785Z Executing ['/opt/conda/bin/python', 'distributed/rpc/test_faulty_agent.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-05-18 03:47:52.842035]
2022-05-18T03:47:53.3927833Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3esolmdf
2022-05-18T03:47:53.3928280Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3esolmdf/_remote_module_non_scriptable.py
2022-05-18T03:47:53.6385831Z <unittest.suite.TestSuite tests=[<__main__.FaultyFaultyAgentDistAutogradTest testMethod=test_context_cleanup_tensor_with_grad>, <__main__.FaultyFaultyAgentDistAutogradTest testMethod=test_verify_backend_options>]>
2022-05-18T03:47:53.6386704Z test_context_cleanup_tensor_with_grad (__main__.FaultyFaultyAgentDistAutogradTest)
2022-05-18T03:47:53.6387140Z test_verify_backend_options (__main__.FaultyFaultyAgentDistAutogradTest)
2022-05-18T03:47:53.6390921Z <unittest.suite.TestSuite tests=[<__main__.FaultyFaultyAgentRpcTest testMethod=test_builtin_remote_message_dropped_timeout>, <__main__.FaultyFaultyAgentRpcTest testMethod=test_builtin_remote_message_dropped_timeout_to_self>, <__main__.FaultyFaultyAgentRpcTest testMethod=test_check_failed_messages>, <__main__.FaultyFaultyAgentRpcTest testMethod=test_custom_faulty_messages>, <__main__.FaultyFaultyAgentRpcTest testMethod=test_custom_messages_to_delay>, <__main__.FaultyFaultyAgentRpcTest testMethod=test_no_faulty_messages>, <__main__.FaultyFaultyAgentRpcTest testMethod=test_remote_message_builtin_delay_timeout>, <__main__.FaultyFaultyAgentRpcTest testMethod=test_remote_message_builtin_delay_timeout_to_self>, <__main__.FaultyFaultyAgentRpcTest testMethod=test_remote_message_dropped_pickle>, <__main__.FaultyFaultyAgentRpcTest testMethod=test_remote_message_dropped_pickle_to_self>, <__main__.FaultyFaultyAgentRpcTest testMethod=test_remote_message_script_delay_timeout>, <__main__.FaultyFaultyAgentRpcTest testMethod=test_remote_message_script_delay_timeout_to_self>, <__main__.FaultyFaultyAgentRpcTest testMethod=test_rpc_builtin_timeout>, <__main__.FaultyFaultyAgentRpcTest testMethod=test_rpc_script_timeout>, <__main__.FaultyFaultyAgentRpcTest testMethod=test_rref_to_here_timeout>, <__main__.FaultyFaultyAgentRpcTest testMethod=test_udf_remote_message_delay_timeout>, <__main__.FaultyFaultyAgentRpcTest testMethod=test_udf_remote_message_delay_timeout_to_self>, <__main__.FaultyFaultyAgentRpcTest testMethod=test_udf_remote_message_dropped_timeout>, <__main__.FaultyFaultyAgentRpcTest testMethod=test_udf_remote_message_dropped_timeout_to_self>, <__main__.FaultyFaultyAgentRpcTest testMethod=test_verify_backend_options>]>
2022-05-18T03:47:53.6393248Z test_builtin_remote_message_dropped_timeout (__main__.FaultyFaultyAgentRpcTest)
2022-05-18T03:47:53.6393608Z test_builtin_remote_message_dropped_timeout_to_self (__main__.FaultyFaultyAgentRpcTest)
2022-05-18T03:47:53.6393923Z test_check_failed_messages (__main__.FaultyFaultyAgentRpcTest)
2022-05-18T03:47:53.6394222Z test_custom_faulty_messages (__main__.FaultyFaultyAgentRpcTest)
2022-05-18T03:47:53.6394522Z test_custom_messages_to_delay (__main__.FaultyFaultyAgentRpcTest)
2022-05-18T03:47:53.6394825Z test_no_faulty_messages (__main__.FaultyFaultyAgentRpcTest)
2022-05-18T03:47:53.6395132Z test_remote_message_builtin_delay_timeout (__main__.FaultyFaultyAgentRpcTest)
2022-05-18T03:47:53.6395473Z test_remote_message_builtin_delay_timeout_to_self (__main__.FaultyFaultyAgentRpcTest)
2022-05-18T03:47:53.6395804Z test_remote_message_dropped_pickle (__main__.FaultyFaultyAgentRpcTest)
2022-05-18T03:47:53.6396115Z test_remote_message_dropped_pickle_to_self (__main__.FaultyFaultyAgentRpcTest)
2022-05-18T03:47:53.6396440Z test_remote_message_script_delay_timeout (__main__.FaultyFaultyAgentRpcTest)
2022-05-18T03:47:53.6396780Z test_remote_message_script_delay_timeout_to_self (__main__.FaultyFaultyAgentRpcTest)
2022-05-18T03:47:53.6397084Z test_rpc_builtin_timeout (__main__.FaultyFaultyAgentRpcTest)
2022-05-18T03:47:53.6397372Z test_rpc_script_timeout (__main__.FaultyFaultyAgentRpcTest)
2022-05-18T03:47:53.6397660Z test_rref_to_here_timeout (__main__.FaultyFaultyAgentRpcTest)
2022-05-18T03:47:53.6397964Z test_udf_remote_message_delay_timeout (__main__.FaultyFaultyAgentRpcTest)
2022-05-18T03:47:53.6398275Z test_udf_remote_message_delay_timeout_to_self (__main__.FaultyFaultyAgentRpcTest)
2022-05-18T03:47:53.6398605Z test_udf_remote_message_dropped_timeout (__main__.FaultyFaultyAgentRpcTest)
2022-05-18T03:47:53.6398938Z test_udf_remote_message_dropped_timeout_to_self (__main__.FaultyFaultyAgentRpcTest)
2022-05-18T03:47:53.6399244Z test_verify_backend_options (__main__.FaultyFaultyAgentRpcTest)
2022-05-18T03:47:53.6400077Z <unittest.suite.TestSuite tests=[<__main__.FaultyJitFaultyAgentRpcTest testMethod=test_remote_timeout_to_here_in_jit>, <__main__.FaultyJitFaultyAgentRpcTest testMethod=test_rref_timeout_pickle_in_jit>, <__main__.FaultyJitFaultyAgentRpcTest testMethod=test_rref_timeout_pickle_script_func>, <__main__.FaultyJitFaultyAgentRpcTest testMethod=test_rref_to_here_timeout_in_jit>, <__main__.FaultyJitFaultyAgentRpcTest testMethod=test_timeout_in_python>, <__main__.FaultyJitFaultyAgentRpcTest testMethod=test_timeout_in_torchscript_function>]>
2022-05-18T03:47:53.6400932Z test_remote_timeout_to_here_in_jit (__main__.FaultyJitFaultyAgentRpcTest)
2022-05-18T03:47:53.6401286Z test_rref_timeout_pickle_in_jit (__main__.FaultyJitFaultyAgentRpcTest)
2022-05-18T03:47:53.6401611Z test_rref_timeout_pickle_script_func (__main__.FaultyJitFaultyAgentRpcTest)
2022-05-18T03:47:53.6401919Z test_rref_to_here_timeout_in_jit (__main__.FaultyJitFaultyAgentRpcTest)
2022-05-18T03:47:53.6402226Z test_timeout_in_python (__main__.FaultyJitFaultyAgentRpcTest)
2022-05-18T03:47:53.6402544Z test_timeout_in_torchscript_function (__main__.FaultyJitFaultyAgentRpcTest)
2022-05-18T03:47:54.1898699Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps_vpuv7t
2022-05-18T03:47:54.1899427Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps_vpuv7t/_remote_module_non_scriptable.py
2022-05-18T03:47:54.4390487Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_faulty_agent
2022-05-18T03:47:54.4400671Z 
2022-05-18T03:47:54.4400916Z Running tests...
2022-05-18T03:47:54.4401578Z ----------------------------------------------------------------------
2022-05-18T03:47:54.7191034Z   test_context_cleanup_tensor_with_grad (__main__.FaultyFaultyAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2097
2022-05-18T03:47:54.7213911Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2098
2022-05-18T03:47:54.7237023Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2099
2022-05-18T03:47:54.7260613Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2100
2022-05-18T03:47:55.3260535Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp42l88irf
2022-05-18T03:47:55.3261374Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp42l88irf/_remote_module_non_scriptable.py
2022-05-18T03:47:55.3323622Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0cx9osg3
2022-05-18T03:47:55.3325197Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0cx9osg3/_remote_module_non_scriptable.py
2022-05-18T03:47:55.3560628Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw4vz4sfp
2022-05-18T03:47:55.3561354Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_7np6jib
2022-05-18T03:47:55.3561959Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw4vz4sfp/_remote_module_non_scriptable.py
2022-05-18T03:47:55.3562613Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_7np6jib/_remote_module_non_scriptable.py
2022-05-18T03:47:55.5737182Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:47:55.5784795Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:47:55.6027885Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:47:55.6031332Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:47:55.6683130Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:47:55.6854025Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:47:55.6886538Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:47:55.6887548Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:47:55.6888839Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:47:55.6890003Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:47:55.6891148Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:47:55.7228940Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:48:03.1409233Z ok (8.701s)
2022-05-18T03:48:03.1409494Z 
2022-05-18T03:48:03.1409851Z ----------------------------------------------------------------------
2022-05-18T03:48:03.1410105Z Ran 1 test in 8.701s
2022-05-18T03:48:03.1410221Z 
2022-05-18T03:48:03.1410283Z OK
2022-05-18T03:48:03.1410361Z 
2022-05-18T03:48:03.1410455Z Generating XML reports...
2022-05-18T03:48:03.1876437Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentDistAutogradTest-20220518034754.xml
2022-05-18T03:48:03.9256803Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_j_70n6s
2022-05-18T03:48:03.9257347Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_j_70n6s/_remote_module_non_scriptable.py
2022-05-18T03:48:04.1706342Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_faulty_agent
2022-05-18T03:48:04.1716639Z 
2022-05-18T03:48:04.1717037Z Running tests...
2022-05-18T03:48:04.1717676Z ----------------------------------------------------------------------
2022-05-18T03:48:04.4513036Z   test_verify_backend_options (__main__.FaultyFaultyAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2284
2022-05-18T03:48:04.4534965Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2285
2022-05-18T03:48:04.4557763Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2286
2022-05-18T03:48:04.4582420Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2287
2022-05-18T03:48:05.1237283Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm3btgus9
2022-05-18T03:48:05.1238365Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm3btgus9/_remote_module_non_scriptable.py
2022-05-18T03:48:05.1559001Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpby1iuf5t
2022-05-18T03:48:05.1559824Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpby1iuf5t/_remote_module_non_scriptable.py
2022-05-18T03:48:05.1700570Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmyz8r8_w
2022-05-18T03:48:05.1702063Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmyz8r8_w/_remote_module_non_scriptable.py
2022-05-18T03:48:05.1895046Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9xny2hn8
2022-05-18T03:48:05.1896476Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9xny2hn8/_remote_module_non_scriptable.py
2022-05-18T03:48:05.3715773Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:48:05.4035115Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:48:05.4139366Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:48:05.4348613Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:48:11.6709208Z ok (7.499s)
2022-05-18T03:48:11.6709464Z 
2022-05-18T03:48:11.6709994Z ----------------------------------------------------------------------
2022-05-18T03:48:11.6710312Z Ran 1 test in 7.499s
2022-05-18T03:48:11.6710428Z 
2022-05-18T03:48:11.6710477Z OK
2022-05-18T03:48:11.6710571Z 
2022-05-18T03:48:11.6710667Z Generating XML reports...
2022-05-18T03:48:11.7175824Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentDistAutogradTest-20220518034804.xml
2022-05-18T03:48:12.4692554Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8kzq1rrr
2022-05-18T03:48:12.4693289Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8kzq1rrr/_remote_module_non_scriptable.py
2022-05-18T03:48:12.7157684Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_faulty_agent
2022-05-18T03:48:12.7166868Z 
2022-05-18T03:48:12.7167258Z Running tests...
2022-05-18T03:48:12.7167847Z ----------------------------------------------------------------------
2022-05-18T03:48:12.9953424Z   test_builtin_remote_message_dropped_timeout (__main__.FaultyFaultyAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2459
2022-05-18T03:48:12.9975481Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2460
2022-05-18T03:48:12.9997902Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2461
2022-05-18T03:48:13.0021374Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2462
2022-05-18T03:48:13.6576710Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxh1xjdn2
2022-05-18T03:48:13.6577516Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxh1xjdn2/_remote_module_non_scriptable.py
2022-05-18T03:48:13.6858552Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppalm5ela
2022-05-18T03:48:13.6859286Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppalm5ela/_remote_module_non_scriptable.py
2022-05-18T03:48:13.6944499Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp81xor234
2022-05-18T03:48:13.6945654Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp81xor234/_remote_module_non_scriptable.py
2022-05-18T03:48:13.7109154Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_aizk64j
2022-05-18T03:48:13.7111046Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_aizk64j/_remote_module_non_scriptable.py
2022-05-18T03:48:13.9046196Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:48:13.9329968Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:48:13.9392866Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:48:13.9577138Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:48:20.2149688Z ok (7.498s)
2022-05-18T03:48:20.2149902Z 
2022-05-18T03:48:20.2150343Z ----------------------------------------------------------------------
2022-05-18T03:48:20.2150656Z Ran 1 test in 7.498s
2022-05-18T03:48:20.2150774Z 
2022-05-18T03:48:20.2150827Z OK
2022-05-18T03:48:20.2150936Z 
2022-05-18T03:48:20.2151077Z Generating XML reports...
2022-05-18T03:48:20.2619624Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20220518034812.xml
2022-05-18T03:48:21.0084718Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp485fwqii
2022-05-18T03:48:21.0085447Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp485fwqii/_remote_module_non_scriptable.py
2022-05-18T03:48:21.2563050Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_faulty_agent
2022-05-18T03:48:21.2573117Z 
2022-05-18T03:48:21.2573405Z Running tests...
2022-05-18T03:48:21.2574053Z ----------------------------------------------------------------------
2022-05-18T03:48:21.5358898Z   test_builtin_remote_message_dropped_timeout_to_self (__main__.FaultyFaultyAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2634
2022-05-18T03:48:21.5381042Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2635
2022-05-18T03:48:21.5403575Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2636
2022-05-18T03:48:21.5427729Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2637
2022-05-18T03:48:22.1084730Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp80fr2xyu
2022-05-18T03:48:22.1085680Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp80fr2xyu/_remote_module_non_scriptable.py
2022-05-18T03:48:22.1466736Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphtn4x7n5
2022-05-18T03:48:22.1467591Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphtn4x7n5/_remote_module_non_scriptable.py
2022-05-18T03:48:22.1683544Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0obmktrv
2022-05-18T03:48:22.1684353Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0obmktrv/_remote_module_non_scriptable.py
2022-05-18T03:48:22.2081323Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplu1_dj6j
2022-05-18T03:48:22.2082457Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplu1_dj6j/_remote_module_non_scriptable.py
2022-05-18T03:48:22.3587741Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:48:22.3979711Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:48:22.4193128Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:48:22.4595134Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:48:28.7554137Z ok (7.498s)
2022-05-18T03:48:28.7554480Z 
2022-05-18T03:48:28.7554905Z ----------------------------------------------------------------------
2022-05-18T03:48:28.7555161Z Ran 1 test in 7.498s
2022-05-18T03:48:28.7555277Z 
2022-05-18T03:48:28.7555343Z OK
2022-05-18T03:48:28.7555437Z 
2022-05-18T03:48:28.7555518Z Generating XML reports...
2022-05-18T03:48:28.8029956Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20220518034821.xml
2022-05-18T03:48:29.5432045Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo6jgvkib
2022-05-18T03:48:29.5432723Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo6jgvkib/_remote_module_non_scriptable.py
2022-05-18T03:48:29.7908743Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_faulty_agent
2022-05-18T03:48:29.7918776Z 
2022-05-18T03:48:29.7918894Z Running tests...
2022-05-18T03:48:29.7919612Z ----------------------------------------------------------------------
2022-05-18T03:48:30.0729580Z   test_check_failed_messages (__main__.FaultyFaultyAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2809
2022-05-18T03:48:30.0752735Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2810
2022-05-18T03:48:30.0775747Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2811
2022-05-18T03:48:30.0800319Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2812
2022-05-18T03:48:30.7087230Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnrpdwpy_
2022-05-18T03:48:30.7088047Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnrpdwpy_/_remote_module_non_scriptable.py
2022-05-18T03:48:30.7191218Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo_2zy74t
2022-05-18T03:48:30.7192379Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo_2zy74t/_remote_module_non_scriptable.py
2022-05-18T03:48:30.7289231Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp_pppkji
2022-05-18T03:48:30.7290273Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp_pppkji/_remote_module_non_scriptable.py
2022-05-18T03:48:30.7326389Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc0dr15w0
2022-05-18T03:48:30.7327679Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc0dr15w0/_remote_module_non_scriptable.py
2022-05-18T03:48:30.9551463Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:48:30.9647395Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:48:30.9751179Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:48:30.9816159Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:48:41.8992633Z ok (12.107s)
2022-05-18T03:48:41.8992790Z 
2022-05-18T03:48:41.8993186Z ----------------------------------------------------------------------
2022-05-18T03:48:41.8994144Z Ran 1 test in 12.107s
2022-05-18T03:48:41.8994295Z 
2022-05-18T03:48:41.8994434Z OK
2022-05-18T03:48:41.8994577Z 
2022-05-18T03:48:41.8994659Z Generating XML reports...
2022-05-18T03:48:41.9489263Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20220518034829.xml
2022-05-18T03:48:42.7177000Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk0nckcpp
2022-05-18T03:48:42.7177812Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk0nckcpp/_remote_module_non_scriptable.py
2022-05-18T03:48:42.9663578Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_faulty_agent
2022-05-18T03:48:42.9672878Z 
2022-05-18T03:48:42.9673015Z Running tests...
2022-05-18T03:48:42.9673608Z ----------------------------------------------------------------------
2022-05-18T03:48:43.2500451Z   test_custom_faulty_messages (__main__.FaultyFaultyAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2984
2022-05-18T03:48:43.2523870Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2985
2022-05-18T03:48:43.2546406Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2986
2022-05-18T03:48:43.2570628Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2987
2022-05-18T03:48:43.9225650Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnqlwhs4z
2022-05-18T03:48:43.9226719Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnqlwhs4z/_remote_module_non_scriptable.py
2022-05-18T03:48:43.9297976Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9ggaeqgi
2022-05-18T03:48:43.9298869Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9ggaeqgi/_remote_module_non_scriptable.py
2022-05-18T03:48:43.9404250Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdd25mb6_
2022-05-18T03:48:43.9405540Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdd25mb6_/_remote_module_non_scriptable.py
2022-05-18T03:48:43.9865921Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm14z3gmv
2022-05-18T03:48:43.9866694Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm14z3gmv/_remote_module_non_scriptable.py
2022-05-18T03:48:44.1689209Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:48:44.1770727Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:48:44.1886009Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:48:44.2337062Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:48:50.4698878Z ok (7.502s)
2022-05-18T03:48:50.4699136Z 
2022-05-18T03:48:50.4699562Z ----------------------------------------------------------------------
2022-05-18T03:48:50.4699818Z Ran 1 test in 7.502s
2022-05-18T03:48:50.4699934Z 
2022-05-18T03:48:50.4699995Z OK
2022-05-18T03:48:50.4700089Z 
2022-05-18T03:48:50.4700186Z Generating XML reports...
2022-05-18T03:48:50.5171512Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20220518034842.xml
2022-05-18T03:48:51.2622565Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz40mvq_w
2022-05-18T03:48:51.2623647Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz40mvq_w/_remote_module_non_scriptable.py
2022-05-18T03:48:51.5090288Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_faulty_agent
2022-05-18T03:48:51.5101132Z 
2022-05-18T03:48:51.5101253Z Running tests...
2022-05-18T03:48:51.5101869Z ----------------------------------------------------------------------
2022-05-18T03:48:51.7875812Z   test_custom_messages_to_delay (__main__.FaultyFaultyAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3159
2022-05-18T03:48:51.7897975Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3160
2022-05-18T03:48:51.7920358Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3161
2022-05-18T03:48:51.7944707Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3162
2022-05-18T03:48:52.4086573Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbhtevgip
2022-05-18T03:48:52.4087462Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbhtevgip/_remote_module_non_scriptable.py
2022-05-18T03:48:52.4310103Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1a04bna_
2022-05-18T03:48:52.4311589Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1a04bna_/_remote_module_non_scriptable.py
2022-05-18T03:48:52.4404550Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9801f83g
2022-05-18T03:48:52.4406425Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9801f83g/_remote_module_non_scriptable.py
2022-05-18T03:48:52.4577758Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkbz618zj
2022-05-18T03:48:52.4578564Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkbz618zj/_remote_module_non_scriptable.py
2022-05-18T03:48:52.6587041Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:48:52.6772759Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:48:52.6856155Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:48:52.7078710Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:48:53.0983463Z ok (1.588s)
2022-05-18T03:48:53.0983675Z 
2022-05-18T03:48:53.0984199Z ----------------------------------------------------------------------
2022-05-18T03:48:53.0984595Z Ran 1 test in 1.588s
2022-05-18T03:48:53.0984700Z 
2022-05-18T03:48:53.0984765Z OK
2022-05-18T03:48:53.0984858Z 
2022-05-18T03:48:53.0984956Z Generating XML reports...
2022-05-18T03:48:53.1486421Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20220518034851.xml
2022-05-18T03:48:53.9004343Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6pz2y0xd
2022-05-18T03:48:53.9005907Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6pz2y0xd/_remote_module_non_scriptable.py
2022-05-18T03:48:54.1498459Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_faulty_agent
2022-05-18T03:48:54.1508685Z 
2022-05-18T03:48:54.1508799Z Running tests...
2022-05-18T03:48:54.1509346Z ----------------------------------------------------------------------
2022-05-18T03:48:54.4306811Z   test_no_faulty_messages (__main__.FaultyFaultyAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3334
2022-05-18T03:48:54.4328876Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3335
2022-05-18T03:48:54.4351807Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3336
2022-05-18T03:48:54.4376059Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3337
2022-05-18T03:48:55.0683677Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0x98xvoq
2022-05-18T03:48:55.0684700Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0x98xvoq/_remote_module_non_scriptable.py
2022-05-18T03:48:55.0754323Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuj_zmxpe
2022-05-18T03:48:55.0755476Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuj_zmxpe/_remote_module_non_scriptable.py
2022-05-18T03:48:55.0847377Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp17fo_3za
2022-05-18T03:48:55.0849957Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp17fo_3za/_remote_module_non_scriptable.py
2022-05-18T03:48:55.0967726Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprtw1oz1f
2022-05-18T03:48:55.0968430Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprtw1oz1f/_remote_module_non_scriptable.py
2022-05-18T03:48:55.3164326Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:48:55.3219526Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:48:55.3325963Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:48:55.3464208Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:49:01.6504134Z ok (7.499s)
2022-05-18T03:49:01.6504343Z 
2022-05-18T03:49:01.6504663Z ----------------------------------------------------------------------
2022-05-18T03:49:01.6504939Z Ran 1 test in 7.499s
2022-05-18T03:49:01.6505130Z 
2022-05-18T03:49:01.6544005Z OK
2022-05-18T03:49:01.6544165Z 
2022-05-18T03:49:01.6544268Z Generating XML reports...
2022-05-18T03:49:01.7002634Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20220518034854.xml
2022-05-18T03:49:02.5185613Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu06fbmng
2022-05-18T03:49:02.5186351Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu06fbmng/_remote_module_non_scriptable.py
2022-05-18T03:49:02.7708442Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_faulty_agent
2022-05-18T03:49:02.7719744Z 
2022-05-18T03:49:02.7719858Z Running tests...
2022-05-18T03:49:02.7720429Z ----------------------------------------------------------------------
2022-05-18T03:49:03.0783502Z   test_remote_message_builtin_delay_timeout (__main__.FaultyFaultyAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3509
2022-05-18T03:49:03.0809325Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3510
2022-05-18T03:49:03.0835609Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3511
2022-05-18T03:49:03.0858296Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3512
2022-05-18T03:49:03.8195136Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvq4cl6tj
2022-05-18T03:49:03.8195918Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvq4cl6tj/_remote_module_non_scriptable.py
2022-05-18T03:49:03.8317083Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptidii1xo
2022-05-18T03:49:03.8318241Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptidii1xo/_remote_module_non_scriptable.py
2022-05-18T03:49:03.9049087Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxfphl6cz
2022-05-18T03:49:03.9050266Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxfphl6cz/_remote_module_non_scriptable.py
2022-05-18T03:49:03.9154099Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptnzwp98t
2022-05-18T03:49:03.9156826Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptnzwp98t/_remote_module_non_scriptable.py
2022-05-18T03:49:04.1038627Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:49:04.1092598Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:49:04.1814904Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:49:04.1959315Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:49:09.6003120Z ok (6.828s)
2022-05-18T03:49:09.6003369Z 
2022-05-18T03:49:09.6003812Z ----------------------------------------------------------------------
2022-05-18T03:49:09.6004203Z Ran 1 test in 6.828s
2022-05-18T03:49:09.6004374Z 
2022-05-18T03:49:09.6004475Z OK
2022-05-18T03:49:09.6004619Z 
2022-05-18T03:49:09.6004767Z Generating XML reports...
2022-05-18T03:49:09.6502391Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20220518034902.xml
2022-05-18T03:49:10.4712219Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb7e44g49
2022-05-18T03:49:10.4713018Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb7e44g49/_remote_module_non_scriptable.py
2022-05-18T03:49:10.7507037Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_faulty_agent
2022-05-18T03:49:10.7517249Z 
2022-05-18T03:49:10.7517359Z Running tests...
2022-05-18T03:49:10.7518085Z ----------------------------------------------------------------------
2022-05-18T03:49:11.0547587Z   test_remote_message_builtin_delay_timeout_to_self (__main__.FaultyFaultyAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3684
2022-05-18T03:49:11.0575179Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3685
2022-05-18T03:49:11.0599323Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3686
2022-05-18T03:49:11.0627219Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3687
2022-05-18T03:49:11.7451869Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprhi35u9h
2022-05-18T03:49:11.7452794Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprhi35u9h/_remote_module_non_scriptable.py
2022-05-18T03:49:11.7525219Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfcm7rlo1
2022-05-18T03:49:11.7526634Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfcm7rlo1/_remote_module_non_scriptable.py
2022-05-18T03:49:11.7719836Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6uqs785d
2022-05-18T03:49:11.7720730Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6uqs785d/_remote_module_non_scriptable.py
2022-05-18T03:49:11.7797798Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpoysqzyxt
2022-05-18T03:49:11.7799258Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpoysqzyxt/_remote_module_non_scriptable.py
2022-05-18T03:49:12.0192393Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:49:12.0361687Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:49:12.0529569Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:49:12.0535370Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:49:14.1164984Z [E request_callback_no_python.cpp:559] Received error while processing request type 260: false INTERNAL ASSERT FAILED at "/var/lib/jenkins/workspace/torch/csrc/distributed/rpc/rref_context.cpp":387, please report a bug to PyTorch. Expected OwnerRRef with id GloballyUniqueId(created_on=0, local_id=0) to be created.
2022-05-18T03:49:14.1166425Z Exception raised from getOwnerRRef at /var/lib/jenkins/workspace/torch/csrc/distributed/rpc/rref_context.cpp:387 (most recent call first):
2022-05-18T03:49:14.1168281Z frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x69 (0x7fabbb7e60d9 in /opt/conda/lib/python3.7/site-packages/torch/lib/libc10.so)
2022-05-18T03:49:14.1170290Z frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) + 0xd2 (0x7fabbb7e23a2 in /opt/conda/lib/python3.7/site-packages/torch/lib/libc10.so)
2022-05-18T03:49:14.1172105Z frame #2: c10::detail::torchInternalAssertFail(char const*, char const*, unsigned int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) + 0x4e (0x7fabbb7e3d3e in /opt/conda/lib/python3.7/site-packages/torch/lib/libc10.so)
2022-05-18T03:49:14.1173782Z frame #3: torch::distributed::rpc::RRefContext::getOwnerRRef(torch::distributed::rpc::GloballyUniqueId const&, bool) + 0x4a4 (0x7fabbf74eb74 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)
2022-05-18T03:49:14.1175987Z frame #4: torch::distributed::rpc::RequestCallbackNoPython::assignOwnerRRef(torch::distributed::rpc::GloballyUniqueId const&, torch::distributed::rpc::GloballyUniqueId const&, c10::intrusive_ptr<c10::ivalue::Future, c10::detail::intrusive_target_default_null_type<c10::ivalue::Future> >) const + 0x70 (0x7fabbf73e400 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)
2022-05-18T03:49:14.1178180Z frame #5: torch::distributed::rpc::RequestCallbackImpl::processScriptRemoteCall(torch::distributed::rpc::RpcCommandBase&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x12a (0x7fabc792510a in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_python.so)
2022-05-18T03:49:14.1180343Z frame #6: torch::distributed::rpc::RequestCallbackNoPython::processRpc(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x14c (0x7fabbf743b5c in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)
2022-05-18T03:49:14.1182534Z frame #7: torch::distributed::rpc::RequestCallbackImpl::processRpcWithErrors(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x65 (0x7fabc7922105 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_python.so)
2022-05-18T03:49:14.1184183Z frame #8: <unknown function> + 0x3d19cfa (0x7fabbf73fcfa in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)
2022-05-18T03:49:14.1185779Z frame #9: torch::distributed::rpc::RequestCallbackNoPython::processMessage(torch::distributed::rpc::Message&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0xa87 (0x7fabbf740ee7 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)
2022-05-18T03:49:14.1187611Z frame #10: torch::distributed::rpc::RequestCallback::operator()(torch::distributed::rpc::Message&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x57 (0x7fabbf737aa7 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)
2022-05-18T03:49:14.1188949Z frame #11: <unknown function> + 0x3d46313 (0x7fabbf76c313 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)
2022-05-18T03:49:14.1190051Z frame #12: c10::ThreadPool::main_loop(unsigned long) + 0x2a3 (0x7fabbb7d6373 in /opt/conda/lib/python3.7/site-packages/torch/lib/libc10.so)
2022-05-18T03:49:14.1190861Z frame #13: <unknown function> + 0xc92bd (0x7fabbb6fb2bd in /opt/conda/lib/libstdc++.so.6)
2022-05-18T03:49:14.1191758Z frame #14: <unknown function> + 0x76ba (0x7fabcfb4f6ba in /lib/x86_64-linux-gnu/libpthread.so.0)
2022-05-18T03:49:14.1192574Z frame #15: clone + 0x6d (0x7fabcf88551d in /lib/x86_64-linux-gnu/libc.so.6)
2022-05-18T03:49:14.1192904Z 
2022-05-18T03:49:14.3707294Z ok (3.619s)
2022-05-18T03:49:14.3707511Z 
2022-05-18T03:49:14.3708002Z ----------------------------------------------------------------------
2022-05-18T03:49:14.3708458Z Ran 1 test in 3.619s
2022-05-18T03:49:14.3708589Z 
2022-05-18T03:49:14.3708652Z OK
2022-05-18T03:49:14.3708901Z 
2022-05-18T03:49:14.3708999Z Generating XML reports...
2022-05-18T03:49:14.4239622Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20220518034910.xml
2022-05-18T03:49:15.2431303Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1mtphdcj
2022-05-18T03:49:15.2431838Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1mtphdcj/_remote_module_non_scriptable.py
2022-05-18T03:49:15.4939759Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_faulty_agent
2022-05-18T03:49:15.4949454Z 
2022-05-18T03:49:15.4949605Z Running tests...
2022-05-18T03:49:15.4950017Z ----------------------------------------------------------------------
2022-05-18T03:49:15.7870396Z   test_remote_message_dropped_pickle (__main__.FaultyFaultyAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3859
2022-05-18T03:49:15.7895138Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3860
2022-05-18T03:49:15.7920605Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3861
2022-05-18T03:49:15.7954542Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3862
2022-05-18T03:49:16.5025301Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp46pacmub
2022-05-18T03:49:16.5026161Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp46pacmub/_remote_module_non_scriptable.py
2022-05-18T03:49:16.5043775Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbzkdlmfu
2022-05-18T03:49:16.5046462Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbzkdlmfu/_remote_module_non_scriptable.py
2022-05-18T03:49:16.5085389Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpggzqfg0_
2022-05-18T03:49:16.5088651Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpggzqfg0_/_remote_module_non_scriptable.py
2022-05-18T03:49:16.5370557Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5foyw0ws
2022-05-18T03:49:16.5371355Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5foyw0ws/_remote_module_non_scriptable.py
2022-05-18T03:49:16.7724093Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:49:16.7726873Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:49:16.7901228Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:49:16.8170133Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:49:23.1093058Z ok (7.614s)
2022-05-18T03:49:23.1093253Z 
2022-05-18T03:49:23.1093574Z ----------------------------------------------------------------------
2022-05-18T03:49:23.1093844Z Ran 1 test in 7.614s
2022-05-18T03:49:23.1093958Z 
2022-05-18T03:49:23.1094021Z OK
2022-05-18T03:49:23.1094127Z 
2022-05-18T03:49:23.1094209Z Generating XML reports...
2022-05-18T03:49:23.1590235Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20220518034915.xml
2022-05-18T03:49:23.9073879Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqe09mhw3
2022-05-18T03:49:23.9074784Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqe09mhw3/_remote_module_non_scriptable.py
2022-05-18T03:49:24.1557526Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_faulty_agent
2022-05-18T03:49:24.1567608Z 
2022-05-18T03:49:24.1567714Z Running tests...
2022-05-18T03:49:24.1568288Z ----------------------------------------------------------------------
2022-05-18T03:49:24.4376228Z   test_remote_message_dropped_pickle_to_self (__main__.FaultyFaultyAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4034
2022-05-18T03:49:24.4398251Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4035
2022-05-18T03:49:24.4421441Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4036
2022-05-18T03:49:24.4445976Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4037
2022-05-18T03:49:25.0685412Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyptv0871
2022-05-18T03:49:25.0686140Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpivyhr5ra
2022-05-18T03:49:25.0686790Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm6drk9lo
2022-05-18T03:49:25.0687472Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyptv0871/_remote_module_non_scriptable.py
2022-05-18T03:49:25.0690054Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpivyhr5ra/_remote_module_non_scriptable.py
2022-05-18T03:49:25.0690739Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm6drk9lo/_remote_module_non_scriptable.py
2022-05-18T03:49:25.0767473Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphr46py1i
2022-05-18T03:49:25.0768787Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphr46py1i/_remote_module_non_scriptable.py
2022-05-18T03:49:25.3185009Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:49:25.3185616Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:49:25.3185963Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:49:25.3257964Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:49:31.6574467Z ok (7.500s)
2022-05-18T03:49:31.6574744Z 
2022-05-18T03:49:31.6575250Z ----------------------------------------------------------------------
2022-05-18T03:49:31.6575536Z Ran 1 test in 7.501s
2022-05-18T03:49:31.6575638Z 
2022-05-18T03:49:31.6575700Z OK
2022-05-18T03:49:31.6575811Z 
2022-05-18T03:49:31.6575910Z Generating XML reports...
2022-05-18T03:49:31.7057516Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20220518034924.xml
2022-05-18T03:49:32.4464099Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzq49oajb
2022-05-18T03:49:32.4464749Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzq49oajb/_remote_module_non_scriptable.py
2022-05-18T03:49:32.6936456Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_faulty_agent
2022-05-18T03:49:32.6946382Z 
2022-05-18T03:49:32.6946888Z Running tests...
2022-05-18T03:49:32.6947310Z ----------------------------------------------------------------------
2022-05-18T03:49:32.9746299Z   test_remote_message_script_delay_timeout (__main__.FaultyFaultyAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4209
2022-05-18T03:49:32.9768748Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4210
2022-05-18T03:49:32.9792058Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4211
2022-05-18T03:49:32.9815589Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4212
2022-05-18T03:49:33.5957195Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpopbes6vr
2022-05-18T03:49:33.5957941Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpopbes6vr/_remote_module_non_scriptable.py
2022-05-18T03:49:33.6324094Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpruwwhx9s
2022-05-18T03:49:33.6324746Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2kw7cte8
2022-05-18T03:49:33.6328012Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpruwwhx9s/_remote_module_non_scriptable.py
2022-05-18T03:49:33.6328729Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2kw7cte8/_remote_module_non_scriptable.py
2022-05-18T03:49:33.6329541Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp51hif3fm
2022-05-18T03:49:33.6330453Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp51hif3fm/_remote_module_non_scriptable.py
2022-05-18T03:49:33.8473781Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:49:33.8801251Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:49:33.8804091Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:49:33.8817773Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:49:39.1926974Z ok (6.498s)
2022-05-18T03:49:39.1927227Z 
2022-05-18T03:49:39.1927759Z ----------------------------------------------------------------------
2022-05-18T03:49:39.1928195Z Ran 1 test in 6.498s
2022-05-18T03:49:39.1928407Z 
2022-05-18T03:49:39.1928538Z OK
2022-05-18T03:49:39.1928705Z 
2022-05-18T03:49:39.1928868Z Generating XML reports...
2022-05-18T03:49:39.2429119Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20220518034932.xml
2022-05-18T03:49:39.9921457Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1r0085bv
2022-05-18T03:49:39.9922224Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1r0085bv/_remote_module_non_scriptable.py
2022-05-18T03:49:40.2400563Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_faulty_agent
2022-05-18T03:49:40.2410774Z 
2022-05-18T03:49:40.2411074Z Running tests...
2022-05-18T03:49:40.2411497Z ----------------------------------------------------------------------
2022-05-18T03:49:40.5206647Z   test_remote_message_script_delay_timeout_to_self (__main__.FaultyFaultyAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4384
2022-05-18T03:49:40.5229149Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4385
2022-05-18T03:49:40.5251960Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4386
2022-05-18T03:49:40.5276012Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4387
2022-05-18T03:49:41.1814129Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0krli160
2022-05-18T03:49:41.1814892Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0krli160/_remote_module_non_scriptable.py
2022-05-18T03:49:41.2588189Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplekdmrj6
2022-05-18T03:49:41.2588970Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplekdmrj6/_remote_module_non_scriptable.py
2022-05-18T03:49:41.2606833Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpozn1ep_z
2022-05-18T03:49:41.2608432Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpozn1ep_z/_remote_module_non_scriptable.py
2022-05-18T03:49:41.2872707Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgvc0ooar
2022-05-18T03:49:41.2873536Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgvc0ooar/_remote_module_non_scriptable.py
2022-05-18T03:49:41.4339031Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:49:41.5059773Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:49:41.5074856Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:49:41.5358196Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:49:43.6223339Z [E request_callback_no_python.cpp:559] Received error while processing request type 260: false INTERNAL ASSERT FAILED at "/var/lib/jenkins/workspace/torch/csrc/distributed/rpc/rref_context.cpp":387, please report a bug to PyTorch. Expected OwnerRRef with id GloballyUniqueId(created_on=0, local_id=0) to be created.
2022-05-18T03:49:43.6225263Z Exception raised from getOwnerRRef at /var/lib/jenkins/workspace/torch/csrc/distributed/rpc/rref_context.cpp:387 (most recent call first):
2022-05-18T03:49:43.6226960Z frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x69 (0x7fbd47c5e0d9 in /opt/conda/lib/python3.7/site-packages/torch/lib/libc10.so)
2022-05-18T03:49:43.6228666Z frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) + 0xd2 (0x7fbd47c5a3a2 in /opt/conda/lib/python3.7/site-packages/torch/lib/libc10.so)
2022-05-18T03:49:43.6230492Z frame #2: c10::detail::torchInternalAssertFail(char const*, char const*, unsigned int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) + 0x4e (0x7fbd47c5bd3e in /opt/conda/lib/python3.7/site-packages/torch/lib/libc10.so)
2022-05-18T03:49:43.6232199Z frame #3: torch::distributed::rpc::RRefContext::getOwnerRRef(torch::distributed::rpc::GloballyUniqueId const&, bool) + 0x4a4 (0x7fbd4bbc6b74 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)
2022-05-18T03:49:43.6234391Z frame #4: torch::distributed::rpc::RequestCallbackNoPython::assignOwnerRRef(torch::distributed::rpc::GloballyUniqueId const&, torch::distributed::rpc::GloballyUniqueId const&, c10::intrusive_ptr<c10::ivalue::Future, c10::detail::intrusive_target_default_null_type<c10::ivalue::Future> >) const + 0x70 (0x7fbd4bbb6400 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)
2022-05-18T03:49:43.6236614Z frame #5: torch::distributed::rpc::RequestCallbackImpl::processScriptRemoteCall(torch::distributed::rpc::RpcCommandBase&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x12a (0x7fbd53d9d10a in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_python.so)
2022-05-18T03:49:43.6238761Z frame #6: torch::distributed::rpc::RequestCallbackNoPython::processRpc(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x14c (0x7fbd4bbbbb5c in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)
2022-05-18T03:49:43.6240971Z frame #7: torch::distributed::rpc::RequestCallbackImpl::processRpcWithErrors(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x65 (0x7fbd53d9a105 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_python.so)
2022-05-18T03:49:43.6242479Z frame #8: <unknown function> + 0x3d19cfa (0x7fbd4bbb7cfa in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)
2022-05-18T03:49:43.6244095Z frame #9: torch::distributed::rpc::RequestCallbackNoPython::processMessage(torch::distributed::rpc::Message&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0xa87 (0x7fbd4bbb8ee7 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)
2022-05-18T03:49:43.6245945Z frame #10: torch::distributed::rpc::RequestCallback::operator()(torch::distributed::rpc::Message&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x57 (0x7fbd4bbafaa7 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)
2022-05-18T03:49:43.6247247Z frame #11: <unknown function> + 0x3d46313 (0x7fbd4bbe4313 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)
2022-05-18T03:49:43.6248379Z frame #12: c10::ThreadPool::main_loop(unsigned long) + 0x2a3 (0x7fbd47c4e373 in /opt/conda/lib/python3.7/site-packages/torch/lib/libc10.so)
2022-05-18T03:49:43.6249191Z frame #13: <unknown function> + 0xc92bd (0x7fbd47b732bd in /opt/conda/lib/libstdc++.so.6)
2022-05-18T03:49:43.6250092Z frame #14: <unknown function> + 0x76ba (0x7fbd5bfc76ba in /lib/x86_64-linux-gnu/libpthread.so.0)
2022-05-18T03:49:43.6250907Z frame #15: clone + 0x6d (0x7fbd5bcfd51d in /lib/x86_64-linux-gnu/libc.so.6)
2022-05-18T03:49:43.6251328Z 
2022-05-18T03:49:43.8346016Z ok (3.593s)
2022-05-18T03:49:43.8347782Z 
2022-05-18T03:49:43.8348595Z ----------------------------------------------------------------------
2022-05-18T03:49:43.8348910Z Ran 1 test in 3.594s
2022-05-18T03:49:43.8349028Z 
2022-05-18T03:49:43.8349079Z OK
2022-05-18T03:49:43.8349172Z 
2022-05-18T03:49:43.8349269Z Generating XML reports...
2022-05-18T03:49:43.8898555Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20220518034940.xml
2022-05-18T03:49:44.6877178Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbkcnffum
2022-05-18T03:49:44.6877905Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbkcnffum/_remote_module_non_scriptable.py
2022-05-18T03:49:44.9359623Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_faulty_agent
2022-05-18T03:49:44.9370549Z 
2022-05-18T03:49:44.9370783Z Running tests...
2022-05-18T03:49:45.2187894Z ----------------------------------------------------------------------
2022-05-18T03:49:45.2188421Z   test_rpc_builtin_timeout (__main__.FaultyFaultyAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4559
2022-05-18T03:49:45.2212353Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4560
2022-05-18T03:49:45.2235938Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4561
2022-05-18T03:49:45.2260975Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4562
2022-05-18T03:49:45.8309219Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_64aly14
2022-05-18T03:49:45.8309960Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_64aly14/_remote_module_non_scriptable.py
2022-05-18T03:49:45.8406750Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp94caq8fp
2022-05-18T03:49:45.8408313Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp94caq8fp/_remote_module_non_scriptable.py
2022-05-18T03:49:45.8557681Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmrm3q0oe
2022-05-18T03:49:45.8558388Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmrm3q0oe/_remote_module_non_scriptable.py
2022-05-18T03:49:45.8659611Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmxw9q3mk
2022-05-18T03:49:45.8661326Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmxw9q3mk/_remote_module_non_scriptable.py
2022-05-18T03:49:46.0805410Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:49:46.0889255Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:49:46.1033004Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:49:46.1156047Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:49:59.9500994Z ok (15.013s)
2022-05-18T03:49:59.9501210Z 
2022-05-18T03:49:59.9501614Z ----------------------------------------------------------------------
2022-05-18T03:49:59.9501935Z Ran 1 test in 15.013s
2022-05-18T03:49:59.9502051Z 
2022-05-18T03:49:59.9502104Z OK
2022-05-18T03:49:59.9502200Z 
2022-05-18T03:49:59.9502294Z Generating XML reports...
2022-05-18T03:49:59.9976281Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20220518034944.xml
2022-05-18T03:50:00.7372877Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp39vvrlwr
2022-05-18T03:50:00.7373657Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp39vvrlwr/_remote_module_non_scriptable.py
2022-05-18T03:50:00.9823614Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_faulty_agent
2022-05-18T03:50:00.9832984Z 
2022-05-18T03:50:00.9833307Z Running tests...
2022-05-18T03:50:00.9833927Z ----------------------------------------------------------------------
2022-05-18T03:50:01.2631045Z   test_rpc_script_timeout (__main__.FaultyFaultyAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4734
2022-05-18T03:50:01.2653883Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4735
2022-05-18T03:50:01.2677223Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4736
2022-05-18T03:50:01.2702622Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4737
2022-05-18T03:50:01.9344229Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp105a6h5f
2022-05-18T03:50:01.9345352Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp105a6h5f/_remote_module_non_scriptable.py
2022-05-18T03:50:01.9590174Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpoj62neun
2022-05-18T03:50:01.9590986Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpoj62neun/_remote_module_non_scriptable.py
2022-05-18T03:50:02.0036756Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy91zmx64
2022-05-18T03:50:02.0037627Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy91zmx64/_remote_module_non_scriptable.py
2022-05-18T03:50:02.0046528Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe19h0qya
2022-05-18T03:50:02.0048686Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe19h0qya/_remote_module_non_scriptable.py
2022-05-18T03:50:02.1806604Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:50:02.2047594Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:50:02.2497271Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:50:02.2526005Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:50:10.0853284Z ok (9.102s)
2022-05-18T03:50:10.0853514Z 
2022-05-18T03:50:10.0853854Z ----------------------------------------------------------------------
2022-05-18T03:50:10.0854684Z Ran 1 test in 9.102s
2022-05-18T03:50:10.0854870Z 
2022-05-18T03:50:10.0854977Z OK
2022-05-18T03:50:10.0855159Z 
2022-05-18T03:50:10.0855335Z Generating XML reports...
2022-05-18T03:50:10.1343518Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20220518035000.xml
2022-05-18T03:50:10.8866997Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_yxa1gy7
2022-05-18T03:50:10.8867589Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_yxa1gy7/_remote_module_non_scriptable.py
2022-05-18T03:50:11.1326174Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_faulty_agent
2022-05-18T03:50:11.1336076Z 
2022-05-18T03:50:11.1336305Z Running tests...
2022-05-18T03:50:11.1336936Z ----------------------------------------------------------------------
2022-05-18T03:50:11.4168002Z   test_rref_to_here_timeout (__main__.FaultyFaultyAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4909
2022-05-18T03:50:11.4191917Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4910
2022-05-18T03:50:11.4215479Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4911
2022-05-18T03:50:11.4241254Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4912
2022-05-18T03:50:12.1123062Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpoe8g07gz
2022-05-18T03:50:12.1123854Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpoe8g07gz/_remote_module_non_scriptable.py
2022-05-18T03:50:12.1296624Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxghrsie4
2022-05-18T03:50:12.1297664Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxghrsie4/_remote_module_non_scriptable.py
2022-05-18T03:50:12.1571864Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxj__bngi
2022-05-18T03:50:12.1572676Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxj__bngi/_remote_module_non_scriptable.py
2022-05-18T03:50:12.1629510Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg7nga0ay
2022-05-18T03:50:12.1630953Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg7nga0ay/_remote_module_non_scriptable.py
2022-05-18T03:50:12.3618507Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:50:12.3785163Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:50:12.4039417Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:50:12.4120668Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:50:14.7310079Z ok (3.597s)
2022-05-18T03:50:14.7310316Z 
2022-05-18T03:50:14.7310788Z ----------------------------------------------------------------------
2022-05-18T03:50:14.7311178Z Ran 1 test in 3.597s
2022-05-18T03:50:14.7311344Z 
2022-05-18T03:50:14.7311434Z OK
2022-05-18T03:50:14.7311576Z 
2022-05-18T03:50:14.7311714Z Generating XML reports...
2022-05-18T03:50:14.7787297Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20220518035011.xml
2022-05-18T03:50:15.5194174Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg1tdc4uf
2022-05-18T03:50:15.5194637Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg1tdc4uf/_remote_module_non_scriptable.py
2022-05-18T03:50:15.7665372Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_faulty_agent
2022-05-18T03:50:15.7674882Z 
2022-05-18T03:50:15.7675366Z Running tests...
2022-05-18T03:50:15.7675809Z ----------------------------------------------------------------------
2022-05-18T03:50:16.0463652Z   test_udf_remote_message_delay_timeout (__main__.FaultyFaultyAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5084
2022-05-18T03:50:16.0486844Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5085
2022-05-18T03:50:16.0509709Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5086
2022-05-18T03:50:16.0534528Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5087
2022-05-18T03:50:16.7055479Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa3af70ce
2022-05-18T03:50:16.7056250Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa3af70ce/_remote_module_non_scriptable.py
2022-05-18T03:50:16.7156925Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps0v3dwyo
2022-05-18T03:50:16.7157679Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps0v3dwyo/_remote_module_non_scriptable.py
2022-05-18T03:50:16.7338247Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6pyel_oa
2022-05-18T03:50:16.7339303Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6pyel_oa/_remote_module_non_scriptable.py
2022-05-18T03:50:16.7473791Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptah00dt0
2022-05-18T03:50:16.7475777Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptah00dt0/_remote_module_non_scriptable.py
2022-05-18T03:50:16.9602997Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:50:16.9636897Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:50:16.9799219Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:50:17.0037038Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:50:23.2662162Z ok (7.498s)
2022-05-18T03:50:23.2662323Z 
2022-05-18T03:50:23.2663077Z ----------------------------------------------------------------------
2022-05-18T03:50:23.2663348Z Ran 1 test in 7.499s
2022-05-18T03:50:23.2663464Z 
2022-05-18T03:50:23.2663527Z OK
2022-05-18T03:50:23.2663625Z 
2022-05-18T03:50:23.2663707Z Generating XML reports...
2022-05-18T03:50:23.3166723Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20220518035015.xml
2022-05-18T03:50:24.0580621Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjmmnr2gi
2022-05-18T03:50:24.0581161Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjmmnr2gi/_remote_module_non_scriptable.py
2022-05-18T03:50:24.3048085Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_faulty_agent
2022-05-18T03:50:24.3058494Z 
2022-05-18T03:50:24.3058837Z Running tests...
2022-05-18T03:50:24.3059234Z ----------------------------------------------------------------------
2022-05-18T03:50:24.5886984Z   test_udf_remote_message_delay_timeout_to_self (__main__.FaultyFaultyAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5259
2022-05-18T03:50:24.5911075Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5260
2022-05-18T03:50:24.5933983Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5261
2022-05-18T03:50:24.5958420Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5262
2022-05-18T03:50:25.2263754Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn06eyvmo
2022-05-18T03:50:25.2264568Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn06eyvmo/_remote_module_non_scriptable.py
2022-05-18T03:50:25.2293707Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpit5icq_5
2022-05-18T03:50:25.2294886Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpit5icq_5/_remote_module_non_scriptable.py
2022-05-18T03:50:25.2445293Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp02shtlyf
2022-05-18T03:50:25.2448413Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp02shtlyf/_remote_module_non_scriptable.py
2022-05-18T03:50:25.2539621Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_4juewl5
2022-05-18T03:50:25.2540464Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_4juewl5/_remote_module_non_scriptable.py
2022-05-18T03:50:25.4743791Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:50:25.4814734Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:50:25.4924292Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:50:25.5004973Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:50:28.5894653Z [E request_callback_no_python.cpp:559] Received error while processing request type 261: false INTERNAL ASSERT FAILED at "/var/lib/jenkins/workspace/torch/csrc/distributed/rpc/rref_context.cpp":387, please report a bug to PyTorch. Expected OwnerRRef with id GloballyUniqueId(created_on=0, local_id=0) to be created.
2022-05-18T03:50:28.5895463Z Exception raised from getOwnerRRef at /var/lib/jenkins/workspace/torch/csrc/distributed/rpc/rref_context.cpp:387 (most recent call first):
2022-05-18T03:50:28.5896731Z frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x69 (0x7f54a4ace0d9 in /opt/conda/lib/python3.7/site-packages/torch/lib/libc10.so)
2022-05-18T03:50:28.5897486Z frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) + 0xd2 (0x7f54a4aca3a2 in /opt/conda/lib/python3.7/site-packages/torch/lib/libc10.so)
2022-05-18T03:50:28.5898578Z frame #2: c10::detail::torchInternalAssertFail(char const*, char const*, unsigned int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) + 0x4e (0x7f54a4acbd3e in /opt/conda/lib/python3.7/site-packages/torch/lib/libc10.so)
2022-05-18T03:50:28.5899912Z frame #3: torch::distributed::rpc::RRefContext::getOwnerRRef(torch::distributed::rpc::GloballyUniqueId const&, bool) + 0x4a4 (0x7f54a8a36b74 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)
2022-05-18T03:50:28.5900906Z frame #4: torch::distributed::rpc::RequestCallbackNoPython::assignOwnerRRef(torch::distributed::rpc::GloballyUniqueId const&, torch::distributed::rpc::GloballyUniqueId const&, c10::intrusive_ptr<c10::ivalue::Future, c10::detail::intrusive_target_default_null_type<c10::ivalue::Future> >) const + 0x70 (0x7f54a8a26400 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)
2022-05-18T03:50:28.5901893Z frame #5: torch::distributed::rpc::RequestCallbackImpl::processPythonRemoteCall(torch::distributed::rpc::RpcCommandBase&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0xc8 (0x7f54b0c0aae8 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_python.so)
2022-05-18T03:50:28.5902850Z frame #6: torch::distributed::rpc::RequestCallbackNoPython::processRpc(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x194 (0x7f54a8a2bba4 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)
2022-05-18T03:50:28.5903960Z frame #7: torch::distributed::rpc::RequestCallbackImpl::processRpcWithErrors(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x65 (0x7f54b0c0a105 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_python.so)
2022-05-18T03:50:28.5904631Z frame #8: <unknown function> + 0x3d19cfa (0x7f54a8a27cfa in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)
2022-05-18T03:50:28.5905349Z frame #9: torch::distributed::rpc::RequestCallbackNoPython::processMessage(torch::distributed::rpc::Message&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0xa87 (0x7f54a8a28ee7 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)
2022-05-18T03:50:28.5906151Z frame #10: torch::distributed::rpc::RequestCallback::operator()(torch::distributed::rpc::Message&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x57 (0x7f54a8a1faa7 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)
2022-05-18T03:50:28.5906745Z frame #11: <unknown function> + 0x3d46313 (0x7f54a8a54313 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)
2022-05-18T03:50:28.5907251Z frame #12: c10::ThreadPool::main_loop(unsigned long) + 0x2a3 (0x7f54a4abe373 in /opt/conda/lib/python3.7/site-packages/torch/lib/libc10.so)
2022-05-18T03:50:28.5907617Z frame #13: <unknown function> + 0xc92bd (0x7f54a49e32bd in /opt/conda/lib/libstdc++.so.6)
2022-05-18T03:50:28.5908014Z frame #14: <unknown function> + 0x76ba (0x7f54b8e376ba in /lib/x86_64-linux-gnu/libpthread.so.0)
2022-05-18T03:50:28.5908386Z frame #15: clone + 0x6d (0x7f54b8b6d51d in /lib/x86_64-linux-gnu/libc.so.6)
2022-05-18T03:50:28.5908546Z 
2022-05-18T03:50:28.5908829Z [W tensorpipe_agent.cpp:627] RPC agent for worker0 won't send response to request #0 to worker0, as the agent is shutting down
2022-05-18T03:50:28.7038678Z ok (4.398s)
2022-05-18T03:50:28.7038918Z 
2022-05-18T03:50:28.7039431Z ----------------------------------------------------------------------
2022-05-18T03:50:28.7039906Z Ran 1 test in 4.398s
2022-05-18T03:50:28.7040094Z 
2022-05-18T03:50:28.7040173Z OK
2022-05-18T03:50:28.7040268Z 
2022-05-18T03:50:28.7040353Z Generating XML reports...
2022-05-18T03:50:28.7516278Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20220518035024.xml
2022-05-18T03:50:29.4860441Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_ny_3inw
2022-05-18T03:50:29.4861763Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_ny_3inw/_remote_module_non_scriptable.py
2022-05-18T03:50:29.7318439Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_faulty_agent
2022-05-18T03:50:29.7328356Z 
2022-05-18T03:50:29.7328460Z Running tests...
2022-05-18T03:50:29.7329041Z ----------------------------------------------------------------------
2022-05-18T03:50:30.0101763Z   test_udf_remote_message_dropped_timeout (__main__.FaultyFaultyAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5434
2022-05-18T03:50:30.0125294Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5435
2022-05-18T03:50:30.0148419Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5436
2022-05-18T03:50:30.0172713Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5437
2022-05-18T03:50:30.7002416Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdss5orvk
2022-05-18T03:50:30.7003167Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdss5orvk/_remote_module_non_scriptable.py
2022-05-18T03:50:30.7311612Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkx4u847b
2022-05-18T03:50:30.7312491Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkx4u847b/_remote_module_non_scriptable.py
2022-05-18T03:50:30.7385698Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2qi0w81x
2022-05-18T03:50:30.7387128Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2qi0w81x/_remote_module_non_scriptable.py
2022-05-18T03:50:30.7470218Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm00istez
2022-05-18T03:50:30.7482789Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm00istez/_remote_module_non_scriptable.py
2022-05-18T03:50:30.9485025Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:50:30.9779264Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:50:30.9831551Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:50:30.9934923Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:50:37.2299999Z ok (7.497s)
2022-05-18T03:50:37.2300253Z 
2022-05-18T03:50:37.2300705Z ----------------------------------------------------------------------
2022-05-18T03:50:37.2300954Z Ran 1 test in 7.497s
2022-05-18T03:50:37.2301069Z 
2022-05-18T03:50:37.2301117Z OK
2022-05-18T03:50:37.2301211Z 
2022-05-18T03:50:37.2301311Z Generating XML reports...
2022-05-18T03:50:37.2830220Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20220518035029.xml
2022-05-18T03:50:38.0428189Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpess26p18
2022-05-18T03:50:38.0428834Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpess26p18/_remote_module_non_scriptable.py
2022-05-18T03:50:38.2902580Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_faulty_agent
2022-05-18T03:50:38.2912397Z 
2022-05-18T03:50:38.2912554Z Running tests...
2022-05-18T03:50:38.2913052Z ----------------------------------------------------------------------
2022-05-18T03:50:38.5741505Z   test_udf_remote_message_dropped_timeout_to_self (__main__.FaultyFaultyAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5609
2022-05-18T03:50:38.5764157Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5610
2022-05-18T03:50:38.5787982Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5611
2022-05-18T03:50:38.5812656Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5612
2022-05-18T03:50:39.1802336Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpknpobok1
2022-05-18T03:50:39.1803154Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpknpobok1/_remote_module_non_scriptable.py
2022-05-18T03:50:39.1891383Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfpq_w45b
2022-05-18T03:50:39.1892787Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfpq_w45b/_remote_module_non_scriptable.py
2022-05-18T03:50:39.1929436Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc8wfh1rz
2022-05-18T03:50:39.1931400Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc8wfh1rz/_remote_module_non_scriptable.py
2022-05-18T03:50:39.2078147Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8dqucuiw
2022-05-18T03:50:39.2078959Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8dqucuiw/_remote_module_non_scriptable.py
2022-05-18T03:50:39.4276705Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:50:39.4340142Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:50:39.4395301Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:50:39.4546060Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:50:45.6937261Z ok (7.402s)
2022-05-18T03:50:45.6937517Z 
2022-05-18T03:50:45.6938048Z ----------------------------------------------------------------------
2022-05-18T03:50:45.6938313Z Ran 1 test in 7.402s
2022-05-18T03:50:45.6938436Z 
2022-05-18T03:50:45.6938497Z OK
2022-05-18T03:50:45.6938590Z 
2022-05-18T03:50:45.6938685Z Generating XML reports...
2022-05-18T03:50:45.7421527Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20220518035038.xml
2022-05-18T03:50:46.4874870Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmbv1cl8d
2022-05-18T03:50:46.4875574Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmbv1cl8d/_remote_module_non_scriptable.py
2022-05-18T03:50:46.7349400Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_faulty_agent
2022-05-18T03:50:46.7358876Z 
2022-05-18T03:50:46.7359217Z Running tests...
2022-05-18T03:50:46.7359611Z ----------------------------------------------------------------------
2022-05-18T03:50:47.0162691Z   test_verify_backend_options (__main__.FaultyFaultyAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5784
2022-05-18T03:50:47.0186892Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5785
2022-05-18T03:50:47.0209448Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5786
2022-05-18T03:50:47.0233613Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5787
2022-05-18T03:50:47.6945391Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu5jh62yc
2022-05-18T03:50:47.6946290Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu5jh62yc/_remote_module_non_scriptable.py
2022-05-18T03:50:47.7164362Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7x5vi4z2
2022-05-18T03:50:47.7165520Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7x5vi4z2/_remote_module_non_scriptable.py
2022-05-18T03:50:47.7319481Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy0yfp_l9
2022-05-18T03:50:47.7320647Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy0yfp_l9/_remote_module_non_scriptable.py
2022-05-18T03:50:47.7322462Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm_759qef
2022-05-18T03:50:47.7325373Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm_759qef/_remote_module_non_scriptable.py
2022-05-18T03:50:47.9432082Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:50:47.9618087Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:50:47.9786769Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:50:47.9804942Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:50:54.2364456Z ok (7.500s)
2022-05-18T03:50:54.2422990Z 
2022-05-18T03:50:54.2423795Z ----------------------------------------------------------------------
2022-05-18T03:50:54.2424187Z Ran 1 test in 7.500s
2022-05-18T03:50:54.2424308Z 
2022-05-18T03:50:54.2424364Z OK
2022-05-18T03:50:54.2424458Z 
2022-05-18T03:50:54.2424550Z Generating XML reports...
2022-05-18T03:50:54.2856046Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20220518035046.xml
2022-05-18T03:50:55.0344242Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjwqeyn1b
2022-05-18T03:50:55.0345088Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjwqeyn1b/_remote_module_non_scriptable.py
2022-05-18T03:50:55.2834711Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_faulty_agent
2022-05-18T03:50:55.2843544Z 
2022-05-18T03:50:55.2843671Z Running tests...
2022-05-18T03:50:55.2844385Z ----------------------------------------------------------------------
2022-05-18T03:50:55.5676805Z   test_remote_timeout_to_here_in_jit (__main__.FaultyJitFaultyAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5959
2022-05-18T03:50:55.5699375Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5960
2022-05-18T03:50:55.5723037Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5961
2022-05-18T03:50:55.5748205Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5962
2022-05-18T03:50:56.2523405Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy6mko6p3
2022-05-18T03:50:56.2524416Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy6mko6p3/_remote_module_non_scriptable.py
2022-05-18T03:50:56.3012680Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp627k1p9x
2022-05-18T03:50:56.3013621Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp627k1p9x/_remote_module_non_scriptable.py
2022-05-18T03:50:56.3201551Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd9zjuteb
2022-05-18T03:50:56.3202465Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd9zjuteb/_remote_module_non_scriptable.py
2022-05-18T03:50:56.3290717Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1raft6yl
2022-05-18T03:50:56.3291985Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1raft6yl/_remote_module_non_scriptable.py
2022-05-18T03:50:56.5004808Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:50:56.5478420Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:50:56.5658901Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:50:56.5766048Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:51:02.8876262Z ok (7.603s)
2022-05-18T03:51:02.8876477Z 
2022-05-18T03:51:02.8876994Z ----------------------------------------------------------------------
2022-05-18T03:51:02.8877335Z Ran 1 test in 7.603s
2022-05-18T03:51:02.8877450Z 
2022-05-18T03:51:02.8877510Z OK
2022-05-18T03:51:02.8877601Z 
2022-05-18T03:51:02.8877692Z Generating XML reports...
2022-05-18T03:51:02.9353174Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyJitFaultyAgentRpcTest-20220518035055.xml
2022-05-18T03:51:03.6762374Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2l9cfjnj
2022-05-18T03:51:03.6763119Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2l9cfjnj/_remote_module_non_scriptable.py
2022-05-18T03:51:03.9217343Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_faulty_agent
2022-05-18T03:51:03.9227814Z 
2022-05-18T03:51:03.9228049Z Running tests...
2022-05-18T03:51:03.9228674Z ----------------------------------------------------------------------
2022-05-18T03:51:04.2027585Z   test_rref_timeout_pickle_in_jit (__main__.FaultyJitFaultyAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6134
2022-05-18T03:51:04.2049952Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6135
2022-05-18T03:51:04.2073048Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6136
2022-05-18T03:51:04.2097409Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6137
2022-05-18T03:51:04.7737475Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph7xbh7j0
2022-05-18T03:51:04.7738260Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph7xbh7j0/_remote_module_non_scriptable.py
2022-05-18T03:51:04.7873390Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpte28dts4
2022-05-18T03:51:04.7874363Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpte28dts4/_remote_module_non_scriptable.py
2022-05-18T03:51:04.8264920Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm7abhqsc
2022-05-18T03:51:04.8265702Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm7abhqsc/_remote_module_non_scriptable.py
2022-05-18T03:51:04.8315109Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpphvcndlg
2022-05-18T03:51:04.8316518Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpphvcndlg/_remote_module_non_scriptable.py
2022-05-18T03:51:05.0222382Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:51:05.0355501Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:51:05.0737455Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:51:05.0814496Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:51:11.3223810Z ok (7.399s)
2022-05-18T03:51:11.3224013Z 
2022-05-18T03:51:11.3224451Z ----------------------------------------------------------------------
2022-05-18T03:51:11.3224702Z Ran 1 test in 7.400s
2022-05-18T03:51:11.3224837Z 
2022-05-18T03:51:11.3224932Z OK
2022-05-18T03:51:11.3226486Z 
2022-05-18T03:51:11.3226654Z Generating XML reports...
2022-05-18T03:51:11.3693435Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyJitFaultyAgentRpcTest-20220518035103.xml
2022-05-18T03:51:12.1104255Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5r6ak5oa
2022-05-18T03:51:12.1104773Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5r6ak5oa/_remote_module_non_scriptable.py
2022-05-18T03:51:12.3601273Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_faulty_agent
2022-05-18T03:51:12.3610190Z 
2022-05-18T03:51:12.3610333Z Running tests...
2022-05-18T03:51:12.3610960Z ----------------------------------------------------------------------
2022-05-18T03:51:12.6396561Z   test_rref_timeout_pickle_script_func (__main__.FaultyJitFaultyAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6309
2022-05-18T03:51:12.6420330Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6310
2022-05-18T03:51:12.6444179Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6311
2022-05-18T03:51:12.6468967Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6312
2022-05-18T03:51:13.2172726Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvaatjlq0
2022-05-18T03:51:13.2173983Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvaatjlq0/_remote_module_non_scriptable.py
2022-05-18T03:51:13.2248088Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprwz4kghj
2022-05-18T03:51:13.2249982Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprwz4kghj/_remote_module_non_scriptable.py
2022-05-18T03:51:13.2744169Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9g24133j
2022-05-18T03:51:13.2744983Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9g24133j/_remote_module_non_scriptable.py
2022-05-18T03:51:13.2853417Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1qpuee3e
2022-05-18T03:51:13.2854303Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1qpuee3e/_remote_module_non_scriptable.py
2022-05-18T03:51:13.4658208Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:51:13.4742282Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:51:13.5229604Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:51:13.5315193Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:51:19.7598131Z ok (7.398s)
2022-05-18T03:51:19.7598366Z 
2022-05-18T03:51:19.7598780Z ----------------------------------------------------------------------
2022-05-18T03:51:19.7599050Z Ran 1 test in 7.399s
2022-05-18T03:51:19.7599167Z 
2022-05-18T03:51:19.7599230Z OK
2022-05-18T03:51:19.7599322Z 
2022-05-18T03:51:19.7599404Z Generating XML reports...
2022-05-18T03:51:19.8117037Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyJitFaultyAgentRpcTest-20220518035112.xml
2022-05-18T03:51:20.5739726Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0c5hda9x
2022-05-18T03:51:20.5740225Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0c5hda9x/_remote_module_non_scriptable.py
2022-05-18T03:51:20.8199397Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_faulty_agent
2022-05-18T03:51:20.8208957Z 
2022-05-18T03:51:20.8209076Z Running tests...
2022-05-18T03:51:20.8209456Z ----------------------------------------------------------------------
2022-05-18T03:51:21.1008382Z   test_rref_to_here_timeout_in_jit (__main__.FaultyJitFaultyAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6484
2022-05-18T03:51:21.1031246Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6485
2022-05-18T03:51:21.1054226Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6486
2022-05-18T03:51:21.1079430Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6487
2022-05-18T03:51:21.7070968Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdzmcsp9r
2022-05-18T03:51:21.7071828Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdzmcsp9r/_remote_module_non_scriptable.py
2022-05-18T03:51:21.7382114Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwtjmaeqq
2022-05-18T03:51:21.7383283Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwtjmaeqq/_remote_module_non_scriptable.py
2022-05-18T03:51:21.7464224Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjnfz3kv_
2022-05-18T03:51:21.7465673Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjnfz3kv_/_remote_module_non_scriptable.py
2022-05-18T03:51:21.7647443Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpal7bfypx
2022-05-18T03:51:21.7649054Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpal7bfypx/_remote_module_non_scriptable.py
2022-05-18T03:51:21.9550093Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:51:21.9840354Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:51:21.9941079Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:51:22.0112412Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:51:24.4148968Z ok (3.594s)
2022-05-18T03:51:24.4149173Z 
2022-05-18T03:51:24.4149620Z ----------------------------------------------------------------------
2022-05-18T03:51:24.4150024Z Ran 1 test in 3.594s
2022-05-18T03:51:24.4150208Z 
2022-05-18T03:51:24.4150303Z OK
2022-05-18T03:51:24.4150441Z 
2022-05-18T03:51:24.4150572Z Generating XML reports...
2022-05-18T03:51:24.4627171Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyJitFaultyAgentRpcTest-20220518035120.xml
2022-05-18T03:51:25.2028913Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdm2d_4id
2022-05-18T03:51:25.2030061Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdm2d_4id/_remote_module_non_scriptable.py
2022-05-18T03:51:25.4506745Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_faulty_agent
2022-05-18T03:51:25.4516327Z 
2022-05-18T03:51:25.4516472Z Running tests...
2022-05-18T03:51:25.4516934Z ----------------------------------------------------------------------
2022-05-18T03:51:25.7335458Z   test_timeout_in_python (__main__.FaultyJitFaultyAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6659
2022-05-18T03:51:25.7358517Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6660
2022-05-18T03:51:25.7381652Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6661
2022-05-18T03:51:25.7406062Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6662
2022-05-18T03:51:26.4186917Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppjhso788
2022-05-18T03:51:26.4187709Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppjhso788/_remote_module_non_scriptable.py
2022-05-18T03:51:26.4521968Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw3jpi3az
2022-05-18T03:51:26.4523185Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw3jpi3az/_remote_module_non_scriptable.py
2022-05-18T03:51:26.4816712Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqjod2xy8
2022-05-18T03:51:26.4817588Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqjod2xy8/_remote_module_non_scriptable.py
2022-05-18T03:51:26.4920189Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr9hh4yrp
2022-05-18T03:51:26.4921224Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr9hh4yrp/_remote_module_non_scriptable.py
2022-05-18T03:51:26.6659941Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:51:26.7014240Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:51:26.7277179Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:51:26.7411095Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:51:31.6517219Z ok (6.200s)
2022-05-18T03:51:31.6517483Z 
2022-05-18T03:51:31.6517945Z ----------------------------------------------------------------------
2022-05-18T03:51:31.6518313Z Ran 1 test in 6.200s
2022-05-18T03:51:31.6518429Z 
2022-05-18T03:51:31.6518495Z OK
2022-05-18T03:51:31.6518587Z 
2022-05-18T03:51:31.6518683Z Generating XML reports...
2022-05-18T03:51:31.6990095Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyJitFaultyAgentRpcTest-20220518035125.xml
2022-05-18T03:51:32.4414579Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr0feo851
2022-05-18T03:51:32.4415045Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr0feo851/_remote_module_non_scriptable.py
2022-05-18T03:51:32.6883221Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_faulty_agent
2022-05-18T03:51:32.6892887Z 
2022-05-18T03:51:32.6893001Z Running tests...
2022-05-18T03:51:32.6893592Z ----------------------------------------------------------------------
2022-05-18T03:51:32.9691636Z   test_timeout_in_torchscript_function (__main__.FaultyJitFaultyAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6834
2022-05-18T03:51:32.9715083Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6835
2022-05-18T03:51:32.9737855Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6836
2022-05-18T03:51:32.9762085Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6837
2022-05-18T03:51:33.6472259Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptwhc82wn
2022-05-18T03:51:33.6473363Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptwhc82wn/_remote_module_non_scriptable.py
2022-05-18T03:51:33.6701560Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuwnfpt57
2022-05-18T03:51:33.6703421Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuwnfpt57/_remote_module_non_scriptable.py
2022-05-18T03:51:33.6789483Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbcpkd7je
2022-05-18T03:51:33.6791981Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbcpkd7je/_remote_module_non_scriptable.py
2022-05-18T03:51:33.7009857Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt8iy8ki6
2022-05-18T03:51:33.7010927Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt8iy8ki6/_remote_module_non_scriptable.py
2022-05-18T03:51:33.8970728Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:51:33.9173456Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:51:33.9274875Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:51:33.9462287Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:51:38.8869439Z ok (6.197s)
2022-05-18T03:51:38.8869615Z 
2022-05-18T03:51:38.8870094Z ----------------------------------------------------------------------
2022-05-18T03:51:38.8870543Z Ran 1 test in 6.198s
2022-05-18T03:51:38.8870695Z 
2022-05-18T03:51:38.8870762Z OK
2022-05-18T03:51:38.8870853Z 
2022-05-18T03:51:38.8870966Z Generating XML reports...
2022-05-18T03:51:38.9353508Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyJitFaultyAgentRpcTest-20220518035132.xml
2022-05-18T03:51:39.2311927Z Running distributed/rpc/test_tensorpipe_agent ... [2022-05-18 03:51:39.230781]
2022-05-18T03:51:39.2312942Z Executing ['/opt/conda/bin/python', 'distributed/rpc/test_tensorpipe_agent.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-05-18 03:51:39.230865]
2022-05-18T03:51:39.7981876Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplkyozucg
2022-05-18T03:51:39.7982853Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplkyozucg/_remote_module_non_scriptable.py
2022-05-18T03:51:40.0501295Z <unittest.suite.TestSuite tests=[<__main__.TensorPipeDdpComparisonTest testMethod=test_ddp_comparison>, <__main__.TensorPipeDdpComparisonTest testMethod=test_ddp_comparison_uneven_inputs>, <__main__.TensorPipeDdpComparisonTest testMethod=test_ddp_dist_autograd_local_vs_remote>, <__main__.TensorPipeDdpComparisonTest testMethod=test_ddp_dist_autograd_sparse_grads>]>
2022-05-18T03:51:40.0502747Z test_ddp_comparison (__main__.TensorPipeDdpComparisonTest)
2022-05-18T03:51:40.0503407Z test_ddp_comparison_uneven_inputs (__main__.TensorPipeDdpComparisonTest)
2022-05-18T03:51:40.0503973Z test_ddp_dist_autograd_local_vs_remote (__main__.TensorPipeDdpComparisonTest)
2022-05-18T03:51:40.0504406Z test_ddp_dist_autograd_sparse_grads (__main__.TensorPipeDdpComparisonTest)
2022-05-18T03:51:40.0505276Z <unittest.suite.TestSuite tests=[<__main__.TensorPipeDdpUnderDistAutogradTest testMethod=test_backward_ddp_inside>, <__main__.TensorPipeDdpUnderDistAutogradTest testMethod=test_backward_ddp_outside>, <__main__.TensorPipeDdpUnderDistAutogradTest testMethod=test_backward_ddp_outside_uneven_inputs>, <__main__.TensorPipeDdpUnderDistAutogradTest testMethod=test_backward_no_ddp>]>
2022-05-18T03:51:40.0506365Z test_backward_ddp_inside (__main__.TensorPipeDdpUnderDistAutogradTest)
2022-05-18T03:51:40.0506926Z test_backward_ddp_outside (__main__.TensorPipeDdpUnderDistAutogradTest)
2022-05-18T03:51:40.0507525Z test_backward_ddp_outside_uneven_inputs (__main__.TensorPipeDdpUnderDistAutogradTest)
2022-05-18T03:51:40.0508140Z test_backward_no_ddp (__main__.TensorPipeDdpUnderDistAutogradTest)
2022-05-18T03:51:40.0516871Z <unittest.suite.TestSuite tests=[<__main__.TensorPipeDistAutogradTest testMethod=test_async_dist_autograd>, <__main__.TensorPipeDistAutogradTest testMethod=test_autograd_context>, <__main__.TensorPipeDistAutogradTest testMethod=test_backward_accumulate_grads>, <__main__.TensorPipeDistAutogradTest testMethod=test_backward_autograd_engine_error>, <__main__.TensorPipeDistAutogradTest testMethod=test_backward_complex_python_udf>, <__main__.TensorPipeDistAutogradTest testMethod=test_backward_different_dtypes>, <__main__.TensorPipeDistAutogradTest testMethod=test_backward_different_tensor_dims>, <__main__.TensorPipeDistAutogradTest testMethod=test_backward_invalid_args>, <__main__.TensorPipeDistAutogradTest testMethod=test_backward_multiple_output_tensors>, <__main__.TensorPipeDistAutogradTest testMethod=test_backward_multiple_roots>, <__main__.TensorPipeDistAutogradTest testMethod=test_backward_multiple_round_trips>, <__main__.TensorPipeDistAutogradTest testMethod=test_backward_no_grad_on_tensor>, <__main__.TensorPipeDistAutogradTest testMethod=test_backward_node_failure>, <__main__.TensorPipeDistAutogradTest testMethod=test_backward_node_failure_python_udf>, <__main__.TensorPipeDistAutogradTest testMethod=test_backward_python_udf_error>, <__main__.TensorPipeDistAutogradTest testMethod=test_backward_rref>, <__main__.TensorPipeDistAutogradTest testMethod=test_backward_rref_multi>, <__main__.TensorPipeDistAutogradTest testMethod=test_backward_rref_nested>, <__main__.TensorPipeDistAutogradTest testMethod=test_backward_simple>, <__main__.TensorPipeDistAutogradTest testMethod=test_backward_simple_python_udf>, <__main__.TensorPipeDistAutogradTest testMethod=test_backward_simple_script_call>, <__main__.TensorPipeDistAutogradTest testMethod=test_backward_simple_self>, <__main__.TensorPipeDistAutogradTest testMethod=test_backward_unused_send_function>, <__main__.TensorPipeDistAutogradTest testMethod=test_backward_unused_tensors>, <__main__.TensorPipeDistAutogradTest testMethod=test_backward_verify_hooks>, <__main__.TensorPipeDistAutogradTest testMethod=test_backward_without_context>, <__main__.TensorPipeDistAutogradTest testMethod=test_backward_without_rpc>, <__main__.TensorPipeDistAutogradTest testMethod=test_backwards_nested_python_udf>, <__main__.TensorPipeDistAutogradTest testMethod=test_clean_context_during_backward>, <__main__.TensorPipeDistAutogradTest testMethod=test_context_cleanup_nested_rpc>, <__main__.TensorPipeDistAutogradTest testMethod=test_context_cleanup_no_tensors>, <__main__.TensorPipeDistAutogradTest testMethod=test_context_cleanup_tensor_no_grad>, <__main__.TensorPipeDistAutogradTest testMethod=test_context_cleanup_tensor_with_grad>, <__main__.TensorPipeDistAutogradTest testMethod=test_debug_info>, <__main__.TensorPipeDistAutogradTest testMethod=test_dist_autograd_profiling>, <__main__.TensorPipeDistAutogradTest testMethod=test_error_in_context>, <__main__.TensorPipeDistAutogradTest testMethod=test_grad_copy_sparse_indices_extra_ref>, <__main__.TensorPipeDistAutogradTest testMethod=test_grad_only_on_return_value>, <__main__.TensorPipeDistAutogradTest testMethod=test_grad_only_on_return_value_remote>, <__main__.TensorPipeDistAutogradTest testMethod=test_graph_for_builtin_call>, <__main__.TensorPipeDistAutogradTest testMethod=test_graph_for_builtin_remote_call>, <__main__.TensorPipeDistAutogradTest testMethod=test_graph_for_py_nested_call>, <__main__.TensorPipeDistAutogradTest testMethod=test_graph_for_py_nested_call_itself>, <__main__.TensorPipeDistAutogradTest testMethod=test_graph_for_py_nested_remote_call>, <__main__.TensorPipeDistAutogradTest testMethod=test_graph_for_py_nested_remote_call_itself>, <__main__.TensorPipeDistAutogradTest testMethod=test_graph_for_python_call>, <__main__.TensorPipeDistAutogradTest testMethod=test_graph_for_python_remote_call>, <__main__.TensorPipeDistAutogradTest testMethod=test_mixed_requires_grad>, <__main__.TensorPipeDistAutogradTest testMethod=test_multiple_backward>, <__main__.TensorPipeDistAutogradTest testMethod=test_multiple_backward_with_errors>, <__main__.TensorPipeDistAutogradTest testMethod=test_nested_backward_accumulate_grads>, <__main__.TensorPipeDistAutogradTest testMethod=test_nested_context>, <__main__.TensorPipeDistAutogradTest testMethod=test_no_grad_copy>, <__main__.TensorPipeDistAutogradTest testMethod=test_no_grad_copy_sparse>, <__main__.TensorPipeDistAutogradTest testMethod=test_no_graph_with_tensors_not_require_grad>, <__main__.TensorPipeDistAutogradTest testMethod=test_no_graph_with_tensors_not_require_grad_remote>, <__main__.TensorPipeDistAutogradTest testMethod=test_post_hooks>, <__main__.TensorPipeDistAutogradTest testMethod=test_remote_complex_args>, <__main__.TensorPipeDistAutogradTest testMethod=test_rpc_complex_args>, <__main__.TensorPipeDistAutogradTest testMethod=test_thread_local_context_id>, <__main__.TensorPipeDistAutogradTest testMethod=test_trainer_ps>, <__main__.TensorPipeDistAutogradTest testMethod=test_trainer_ps_torchscript_functions>, <__main__.TensorPipeDistAutogradTest testMethod=test_worker_ids_recorded>]>
2022-05-18T03:51:40.0527818Z test_async_dist_autograd (__main__.TensorPipeDistAutogradTest)
2022-05-18T03:51:40.0528366Z test_autograd_context (__main__.TensorPipeDistAutogradTest)
2022-05-18T03:51:40.0528909Z test_backward_accumulate_grads (__main__.TensorPipeDistAutogradTest)
2022-05-18T03:51:40.0529499Z test_backward_autograd_engine_error (__main__.TensorPipeDistAutogradTest)
2022-05-18T03:51:40.0530061Z test_backward_complex_python_udf (__main__.TensorPipeDistAutogradTest)
2022-05-18T03:51:40.0530653Z test_backward_different_dtypes (__main__.TensorPipeDistAutogradTest)
2022-05-18T03:51:40.0531198Z test_backward_different_tensor_dims (__main__.TensorPipeDistAutogradTest)
2022-05-18T03:51:40.0531769Z test_backward_invalid_args (__main__.TensorPipeDistAutogradTest)
2022-05-18T03:51:40.0532329Z test_backward_multiple_output_tensors (__main__.TensorPipeDistAutogradTest)
2022-05-18T03:51:40.0532878Z test_backward_multiple_roots (__main__.TensorPipeDistAutogradTest)
2022-05-18T03:51:40.0533441Z test_backward_multiple_round_trips (__main__.TensorPipeDistAutogradTest)
2022-05-18T03:51:40.0534029Z test_backward_no_grad_on_tensor (__main__.TensorPipeDistAutogradTest)
2022-05-18T03:51:40.0534569Z test_backward_node_failure (__main__.TensorPipeDistAutogradTest)
2022-05-18T03:51:40.0535127Z test_backward_node_failure_python_udf (__main__.TensorPipeDistAutogradTest)
2022-05-18T03:51:40.0535698Z test_backward_python_udf_error (__main__.TensorPipeDistAutogradTest)
2022-05-18T03:51:40.0536229Z test_backward_rref (__main__.TensorPipeDistAutogradTest)
2022-05-18T03:51:40.0536730Z test_backward_rref_multi (__main__.TensorPipeDistAutogradTest)
2022-05-18T03:51:40.0537249Z test_backward_rref_nested (__main__.TensorPipeDistAutogradTest)
2022-05-18T03:51:40.0537866Z test_backward_simple (__main__.TensorPipeDistAutogradTest)
2022-05-18T03:51:40.0538409Z test_backward_simple_python_udf (__main__.TensorPipeDistAutogradTest)
2022-05-18T03:51:40.0539110Z test_backward_simple_script_call (__main__.TensorPipeDistAutogradTest)
2022-05-18T03:51:40.0539699Z test_backward_simple_self (__main__.TensorPipeDistAutogradTest)
2022-05-18T03:51:40.0540212Z test_backward_unused_send_function (__main__.TensorPipeDistAutogradTest)
2022-05-18T03:51:40.0540514Z test_backward_unused_tensors (__main__.TensorPipeDistAutogradTest)
2022-05-18T03:51:40.0540820Z test_backward_verify_hooks (__main__.TensorPipeDistAutogradTest)
2022-05-18T03:51:40.0541124Z test_backward_without_context (__main__.TensorPipeDistAutogradTest)
2022-05-18T03:51:40.0541417Z test_backward_without_rpc (__main__.TensorPipeDistAutogradTest)
2022-05-18T03:51:40.0541730Z test_backwards_nested_python_udf (__main__.TensorPipeDistAutogradTest)
2022-05-18T03:51:40.0542051Z test_clean_context_during_backward (__main__.TensorPipeDistAutogradTest)
2022-05-18T03:51:40.0542367Z test_context_cleanup_nested_rpc (__main__.TensorPipeDistAutogradTest)
2022-05-18T03:51:40.0542668Z test_context_cleanup_no_tensors (__main__.TensorPipeDistAutogradTest)
2022-05-18T03:51:40.0543148Z test_context_cleanup_tensor_no_grad (__main__.TensorPipeDistAutogradTest)
2022-05-18T03:51:40.0543477Z test_context_cleanup_tensor_with_grad (__main__.TensorPipeDistAutogradTest)
2022-05-18T03:51:40.0543765Z test_debug_info (__main__.TensorPipeDistAutogradTest)
2022-05-18T03:51:40.0544059Z test_dist_autograd_profiling (__main__.TensorPipeDistAutogradTest)
2022-05-18T03:51:40.0544359Z test_error_in_context (__main__.TensorPipeDistAutogradTest)
2022-05-18T03:51:40.0544658Z test_grad_copy_sparse_indices_extra_ref (__main__.TensorPipeDistAutogradTest)
2022-05-18T03:51:40.0544979Z test_grad_only_on_return_value (__main__.TensorPipeDistAutogradTest)
2022-05-18T03:51:40.0545291Z test_grad_only_on_return_value_remote (__main__.TensorPipeDistAutogradTest)
2022-05-18T03:51:40.0545603Z test_graph_for_builtin_call (__main__.TensorPipeDistAutogradTest)
2022-05-18T03:51:40.0545901Z test_graph_for_builtin_remote_call (__main__.TensorPipeDistAutogradTest)
2022-05-18T03:51:40.0546212Z test_graph_for_py_nested_call (__main__.TensorPipeDistAutogradTest)
2022-05-18T03:51:40.0546525Z test_graph_for_py_nested_call_itself (__main__.TensorPipeDistAutogradTest)
2022-05-18T03:51:40.0546830Z test_graph_for_py_nested_remote_call (__main__.TensorPipeDistAutogradTest)
2022-05-18T03:51:40.0547159Z test_graph_for_py_nested_remote_call_itself (__main__.TensorPipeDistAutogradTest)
2022-05-18T03:51:40.0547476Z test_graph_for_python_call (__main__.TensorPipeDistAutogradTest)
2022-05-18T03:51:40.0547785Z test_graph_for_python_remote_call (__main__.TensorPipeDistAutogradTest)
2022-05-18T03:51:40.0548078Z test_mixed_requires_grad (__main__.TensorPipeDistAutogradTest)
2022-05-18T03:51:40.0548371Z test_multiple_backward (__main__.TensorPipeDistAutogradTest)
2022-05-18T03:51:40.0548678Z test_multiple_backward_with_errors (__main__.TensorPipeDistAutogradTest)
2022-05-18T03:51:40.0549161Z test_nested_backward_accumulate_grads (__main__.TensorPipeDistAutogradTest)
2022-05-18T03:51:40.0549471Z test_nested_context (__main__.TensorPipeDistAutogradTest)
2022-05-18T03:51:40.0549753Z test_no_grad_copy (__main__.TensorPipeDistAutogradTest)
2022-05-18T03:51:40.0550029Z test_no_grad_copy_sparse (__main__.TensorPipeDistAutogradTest)
2022-05-18T03:51:40.0550344Z test_no_graph_with_tensors_not_require_grad (__main__.TensorPipeDistAutogradTest)
2022-05-18T03:51:40.0550686Z test_no_graph_with_tensors_not_require_grad_remote (__main__.TensorPipeDistAutogradTest)
2022-05-18T03:51:40.0550995Z test_post_hooks (__main__.TensorPipeDistAutogradTest)
2022-05-18T03:51:40.0551268Z test_remote_complex_args (__main__.TensorPipeDistAutogradTest)
2022-05-18T03:51:40.0551558Z test_rpc_complex_args (__main__.TensorPipeDistAutogradTest)
2022-05-18T03:51:40.0551855Z test_thread_local_context_id (__main__.TensorPipeDistAutogradTest)
2022-05-18T03:51:40.0552221Z test_trainer_ps (__main__.TensorPipeDistAutogradTest)
2022-05-18T03:51:40.0552559Z test_trainer_ps_torchscript_functions (__main__.TensorPipeDistAutogradTest)
2022-05-18T03:51:40.0553145Z test_worker_ids_recorded (__main__.TensorPipeDistAutogradTest)
2022-05-18T03:51:40.0554112Z <unittest.suite.TestSuite tests=[<__main__.TensorPipeDistOptimizerTest testMethod=test_dist_optim>, <__main__.TensorPipeDistOptimizerTest testMethod=test_dist_optim_exception>, <__main__.TensorPipeDistOptimizerTest testMethod=test_dist_optim_exception_on_constructor>, <__main__.TensorPipeDistOptimizerTest testMethod=test_dist_optim_none_grads>]>
2022-05-18T03:51:40.0554718Z test_dist_optim (__main__.TensorPipeDistOptimizerTest)
2022-05-18T03:51:40.0555015Z test_dist_optim_exception (__main__.TensorPipeDistOptimizerTest)
2022-05-18T03:51:40.0555340Z test_dist_optim_exception_on_constructor (__main__.TensorPipeDistOptimizerTest)
2022-05-18T03:51:40.0555663Z test_dist_optim_none_grads (__main__.TensorPipeDistOptimizerTest)
2022-05-18T03:51:40.0556314Z <unittest.suite.TestSuite tests=[<__main__.TensorPipeJitDistAutogradTest testMethod=test_dist_backward>, <__main__.TensorPipeJitDistAutogradTest testMethod=test_get_gradients>, <__main__.TensorPipeJitDistAutogradTest testMethod=test_jit_fork_within_context>, <__main__.TensorPipeJitDistAutogradTest testMethod=test_restore_context_after_swtich_to_jit_thread>]>
2022-05-18T03:51:40.0556944Z test_dist_backward (__main__.TensorPipeJitDistAutogradTest)
2022-05-18T03:51:40.0557243Z test_get_gradients (__main__.TensorPipeJitDistAutogradTest)
2022-05-18T03:51:40.0557548Z test_jit_fork_within_context (__main__.TensorPipeJitDistAutogradTest)
2022-05-18T03:51:40.0557872Z test_restore_context_after_swtich_to_jit_thread (__main__.TensorPipeJitDistAutogradTest)
2022-05-18T03:51:40.0562859Z <unittest.suite.TestSuite tests=[<__main__.TensorPipeJitRpcTest testMethod=test_add_done_callback>, <__main__.TensorPipeJitRpcTest testMethod=test_all_kwargs_are_populated_by_defaults>, <__main__.TensorPipeJitRpcTest testMethod=test_args_and_kwargs_contain_different_types>, <__main__.TensorPipeJitRpcTest testMethod=test_args_kwargs_are_neither_passed>, <__main__.TensorPipeJitRpcTest testMethod=test_async_function_remote>, <__main__.TensorPipeJitRpcTest testMethod=test_async_function_remote_multi>, <__main__.TensorPipeJitRpcTest testMethod=test_async_function_simple>, <__main__.TensorPipeJitRpcTest testMethod=test_async_function_wrong_decorator_order>, <__main__.TensorPipeJitRpcTest testMethod=test_async_function_wrong_return_type>, <__main__.TensorPipeJitRpcTest testMethod=test_async_function_wrong_return_type_remote>, <__main__.TensorPipeJitRpcTest testMethod=test_async_script_throw>, <__main__.TensorPipeJitRpcTest testMethod=test_async_script_udf>, <__main__.TensorPipeJitRpcTest testMethod=test_call_fork_in_jit_with_profiling>, <__main__.TensorPipeJitRpcTest testMethod=test_call_python_function_remotely_from_script_not_supported>, <__main__.TensorPipeJitRpcTest testMethod=test_call_rpc_with_profiling>, <__main__.TensorPipeJitRpcTest testMethod=test_call_script_function_that_not_exists_remotely_from_script>, <__main__.TensorPipeJitRpcTest testMethod=test_call_script_function_that_raises_remotely_from_script>, <__main__.TensorPipeJitRpcTest testMethod=test_callback_chain>, <__main__.TensorPipeJitRpcTest testMethod=test_callback_simple>, <__main__.TensorPipeJitRpcTest testMethod=test_callback_with_exception>, <__main__.TensorPipeJitRpcTest testMethod=test_create_local_script_class_rref_in_py>, <__main__.TensorPipeJitRpcTest testMethod=test_create_local_script_module_rref_in_py>, <__main__.TensorPipeJitRpcTest testMethod=test_create_script_module_on_remote>, <__main__.TensorPipeJitRpcTest testMethod=test_future_passed_between_python_and_jit>, <__main__.TensorPipeJitRpcTest testMethod=test_future_python_annotation>, <__main__.TensorPipeJitRpcTest testMethod=test_kwargs_not_passed>, <__main__.TensorPipeJitRpcTest testMethod=test_less_than_needed_args_are_specified>, <__main__.TensorPipeJitRpcTest testMethod=test_load_script_module_with_pickled_rref>, <__main__.TensorPipeJitRpcTest testMethod=test_local_rref_local_value>, <__main__.TensorPipeJitRpcTest testMethod=test_more_than_needed_args_are_specified>, <__main__.TensorPipeJitRpcTest testMethod=test_my_script_module_with_rrefs>, <__main__.TensorPipeJitRpcTest testMethod=test_no_kwargs_are_populated_by_defaults>, <__main__.TensorPipeJitRpcTest testMethod=test_record_function_jit_end_callbacks_with_fork>, <__main__.TensorPipeJitRpcTest testMethod=test_record_function_on_caller_rpc_async>, <__main__.TensorPipeJitRpcTest testMethod=test_remote_script_module>, <__main__.TensorPipeJitRpcTest testMethod=test_remote_script_throw>, <__main__.TensorPipeJitRpcTest testMethod=test_remote_script_udf>, <__main__.TensorPipeJitRpcTest testMethod=test_return_local_script_class_rref_in_py_and_use_in_script>, <__main__.TensorPipeJitRpcTest testMethod=test_return_local_script_module_rref_in_py_and_use_in_script>, <__main__.TensorPipeJitRpcTest testMethod=test_rpc_async_jit_profiled>, <__main__.TensorPipeJitRpcTest testMethod=test_rpc_torchscript_record_function>, <__main__.TensorPipeJitRpcTest testMethod=test_rref_as_arg_and_return>, <__main__.TensorPipeJitRpcTest testMethod=test_rref_is_owner>, <__main__.TensorPipeJitRpcTest testMethod=test_rref_jit_pickle_not_supported>, <__main__.TensorPipeJitRpcTest testMethod=test_rref_list_mutate>, <__main__.TensorPipeJitRpcTest testMethod=test_rref_local_value>, <__main__.TensorPipeJitRpcTest testMethod=test_rref_python_annotation>, <__main__.TensorPipeJitRpcTest testMethod=test_some_kwargs_are_populated_by_defaults>, <__main__.TensorPipeJitRpcTest testMethod=test_torchscript_function>, <__main__.TensorPipeJitRpcTest testMethod=test_torchscript_function_exception>, <__main__.TensorPipeJitRpcTest testMethod=test_torchscript_functions_not_supported>, <__main__.TensorPipeJitRpcTest testMethod=test_unexepected_kwarg_is_specified>, <__main__.TensorPipeJitRpcTest testMethod=test_user_rrefs_confirmed>, <__main__.TensorPipeJitRpcTest testMethod=test_user_rrefs_confirmed_remote>]>
2022-05-18T03:51:40.0567671Z test_add_done_callback (__main__.TensorPipeJitRpcTest)
2022-05-18T03:51:40.0567977Z test_all_kwargs_are_populated_by_defaults (__main__.TensorPipeJitRpcTest)
2022-05-18T03:51:40.0568286Z test_args_and_kwargs_contain_different_types (__main__.TensorPipeJitRpcTest)
2022-05-18T03:51:40.0568597Z test_args_kwargs_are_neither_passed (__main__.TensorPipeJitRpcTest)
2022-05-18T03:51:40.0568889Z test_async_function_remote (__main__.TensorPipeJitRpcTest)
2022-05-18T03:51:40.0569160Z test_async_function_remote_multi (__main__.TensorPipeJitRpcTest)
2022-05-18T03:51:40.0569446Z test_async_function_simple (__main__.TensorPipeJitRpcTest)
2022-05-18T03:51:40.0569740Z test_async_function_wrong_decorator_order (__main__.TensorPipeJitRpcTest)
2022-05-18T03:51:40.0570052Z test_async_function_wrong_return_type (__main__.TensorPipeJitRpcTest)
2022-05-18T03:51:40.0570351Z test_async_function_wrong_return_type_remote (__main__.TensorPipeJitRpcTest)
2022-05-18T03:51:40.0570648Z test_async_script_throw (__main__.TensorPipeJitRpcTest)
2022-05-18T03:51:40.0570918Z test_async_script_udf (__main__.TensorPipeJitRpcTest)
2022-05-18T03:51:40.0571192Z test_call_fork_in_jit_with_profiling (__main__.TensorPipeJitRpcTest)
2022-05-18T03:51:40.0571523Z test_call_python_function_remotely_from_script_not_supported (__main__.TensorPipeJitRpcTest)
2022-05-18T03:51:40.0571844Z test_call_rpc_with_profiling (__main__.TensorPipeJitRpcTest)
2022-05-18T03:51:40.0572153Z test_call_script_function_that_not_exists_remotely_from_script (__main__.TensorPipeJitRpcTest)
2022-05-18T03:51:40.0572504Z test_call_script_function_that_raises_remotely_from_script (__main__.TensorPipeJitRpcTest)
2022-05-18T03:51:40.0572806Z test_callback_chain (__main__.TensorPipeJitRpcTest)
2022-05-18T03:51:40.0573069Z test_callback_simple (__main__.TensorPipeJitRpcTest)
2022-05-18T03:51:40.0573333Z test_callback_with_exception (__main__.TensorPipeJitRpcTest)
2022-05-18T03:51:40.0573628Z test_create_local_script_class_rref_in_py (__main__.TensorPipeJitRpcTest)
2022-05-18T03:51:40.0573972Z test_create_local_script_module_rref_in_py (__main__.TensorPipeJitRpcTest)
2022-05-18T03:51:40.0574298Z test_create_script_module_on_remote (__main__.TensorPipeJitRpcTest)
2022-05-18T03:51:40.0574606Z test_future_passed_between_python_and_jit (__main__.TensorPipeJitRpcTest)
2022-05-18T03:51:40.0574906Z test_future_python_annotation (__main__.TensorPipeJitRpcTest)
2022-05-18T03:51:40.0575185Z test_kwargs_not_passed (__main__.TensorPipeJitRpcTest)
2022-05-18T03:51:40.0575462Z test_less_than_needed_args_are_specified (__main__.TensorPipeJitRpcTest)
2022-05-18T03:51:40.0575772Z test_load_script_module_with_pickled_rref (__main__.TensorPipeJitRpcTest)
2022-05-18T03:51:40.0576067Z test_local_rref_local_value (__main__.TensorPipeJitRpcTest)
2022-05-18T03:51:40.0576346Z test_more_than_needed_args_are_specified (__main__.TensorPipeJitRpcTest)
2022-05-18T03:51:40.0576648Z test_my_script_module_with_rrefs (__main__.TensorPipeJitRpcTest)
2022-05-18T03:51:40.0576956Z test_no_kwargs_are_populated_by_defaults (__main__.TensorPipeJitRpcTest)
2022-05-18T03:51:40.0577259Z test_record_function_jit_end_callbacks_with_fork (__main__.TensorPipeJitRpcTest)
2022-05-18T03:51:40.0577577Z test_record_function_on_caller_rpc_async (__main__.TensorPipeJitRpcTest)
2022-05-18T03:51:40.0577876Z test_remote_script_module (__main__.TensorPipeJitRpcTest)
2022-05-18T03:51:40.0578147Z test_remote_script_throw (__main__.TensorPipeJitRpcTest)
2022-05-18T03:51:40.0578404Z test_remote_script_udf (__main__.TensorPipeJitRpcTest)
2022-05-18T03:51:40.0578713Z test_return_local_script_class_rref_in_py_and_use_in_script (__main__.TensorPipeJitRpcTest)
2022-05-18T03:51:40.0579140Z test_return_local_script_module_rref_in_py_and_use_in_script (__main__.TensorPipeJitRpcTest)
2022-05-18T03:51:40.0579444Z test_rpc_async_jit_profiled (__main__.TensorPipeJitRpcTest)
2022-05-18T03:51:40.0579744Z test_rpc_torchscript_record_function (__main__.TensorPipeJitRpcTest)
2022-05-18T03:51:40.0580044Z test_rref_as_arg_and_return (__main__.TensorPipeJitRpcTest)
2022-05-18T03:51:40.0580314Z test_rref_is_owner (__main__.TensorPipeJitRpcTest)
2022-05-18T03:51:40.0580584Z test_rref_jit_pickle_not_supported (__main__.TensorPipeJitRpcTest)
2022-05-18T03:51:40.0580868Z test_rref_list_mutate (__main__.TensorPipeJitRpcTest)
2022-05-18T03:51:40.0581137Z test_rref_local_value (__main__.TensorPipeJitRpcTest)
2022-05-18T03:51:40.0581393Z test_rref_python_annotation (__main__.TensorPipeJitRpcTest)
2022-05-18T03:51:40.0581692Z test_some_kwargs_are_populated_by_defaults (__main__.TensorPipeJitRpcTest)
2022-05-18T03:51:40.0581992Z test_torchscript_function (__main__.TensorPipeJitRpcTest)
2022-05-18T03:51:40.0582271Z test_torchscript_function_exception (__main__.TensorPipeJitRpcTest)
2022-05-18T03:51:40.0582584Z test_torchscript_functions_not_supported (__main__.TensorPipeJitRpcTest)
2022-05-18T03:51:40.0591789Z test_unexepected_kwarg_is_specified (__main__.TensorPipeJitRpcTest)
2022-05-18T03:51:40.0592239Z test_user_rrefs_confirmed (__main__.TensorPipeJitRpcTest)
2022-05-18T03:51:40.0592523Z test_user_rrefs_confirmed_remote (__main__.TensorPipeJitRpcTest)
2022-05-18T03:51:40.0592919Z <unittest.suite.TestSuite tests=[<__main__.TensorPipeParameterServerTest testMethod=test_batch_updating_parameter_server>]>
2022-05-18T03:51:40.0593327Z test_batch_updating_parameter_server (__main__.TensorPipeParameterServerTest)
2022-05-18T03:51:40.0593716Z <unittest.suite.TestSuite tests=[<__main__.TensorPipeReinforcementLearningRpcTest testMethod=test_rl_rpc>]>
2022-05-18T03:51:40.0594104Z test_rl_rpc (__main__.TensorPipeReinforcementLearningRpcTest)
2022-05-18T03:51:40.0595682Z <unittest.suite.TestSuite tests=[<__main__.TensorPipeRemoteModuleTest testMethod=test_bad_module>, <__main__.TensorPipeRemoteModuleTest testMethod=test_forward_async>, <__main__.TensorPipeRemoteModuleTest testMethod=test_forward_async_script>, <__main__.TensorPipeRemoteModuleTest testMethod=test_forward_sync>, <__main__.TensorPipeRemoteModuleTest testMethod=test_forward_sync_script>, <__main__.TensorPipeRemoteModuleTest testMethod=test_forward_with_kwargs>, <__main__.TensorPipeRemoteModuleTest testMethod=test_get_module_rref>, <__main__.TensorPipeRemoteModuleTest testMethod=test_remote_module_py_pickle_not_supported>, <__main__.TensorPipeRemoteModuleTest testMethod=test_remote_module_py_pickle_not_supported_script>, <__main__.TensorPipeRemoteModuleTest testMethod=test_remote_parameters>, <__main__.TensorPipeRemoteModuleTest testMethod=test_send_remote_module_with_a_new_attribute_not_pickled_over_the_wire>, <__main__.TensorPipeRemoteModuleTest testMethod=test_train_eval>, <__main__.TensorPipeRemoteModuleTest testMethod=test_unsupported_methods>]>
2022-05-18T03:51:40.0597234Z test_bad_module (__main__.TensorPipeRemoteModuleTest)
2022-05-18T03:51:40.0597513Z test_forward_async (__main__.TensorPipeRemoteModuleTest)
2022-05-18T03:51:40.0597812Z test_forward_async_script (__main__.TensorPipeRemoteModuleTest)
2022-05-18T03:51:40.0598107Z test_forward_sync (__main__.TensorPipeRemoteModuleTest)
2022-05-18T03:51:40.0598389Z test_forward_sync_script (__main__.TensorPipeRemoteModuleTest)
2022-05-18T03:51:40.0598689Z test_forward_with_kwargs (__main__.TensorPipeRemoteModuleTest)
2022-05-18T03:51:40.0598988Z test_get_module_rref (__main__.TensorPipeRemoteModuleTest)
2022-05-18T03:51:40.0599293Z test_remote_module_py_pickle_not_supported (__main__.TensorPipeRemoteModuleTest)
2022-05-18T03:51:40.0599646Z test_remote_module_py_pickle_not_supported_script (__main__.TensorPipeRemoteModuleTest)
2022-05-18T03:51:40.0599971Z test_remote_parameters (__main__.TensorPipeRemoteModuleTest)
2022-05-18T03:51:40.0600320Z test_send_remote_module_with_a_new_attribute_not_pickled_over_the_wire (__main__.TensorPipeRemoteModuleTest)
2022-05-18T03:51:40.0600645Z test_train_eval (__main__.TensorPipeRemoteModuleTest)
2022-05-18T03:51:40.0600935Z test_unsupported_methods (__main__.TensorPipeRemoteModuleTest)
2022-05-18T03:51:40.0618885Z <unittest.suite.TestSuite tests=[<__main__.TensorPipeRpcTest testMethod=test_add>, <__main__.TensorPipeRpcTest testMethod=test_add_done_callback>, <__main__.TensorPipeRpcTest testMethod=test_add_with_id>, <__main__.TensorPipeRpcTest testMethod=test_all_gather>, <__main__.TensorPipeRpcTest testMethod=test_all_gather_timeout>, <__main__.TensorPipeRpcTest testMethod=test_async_add>, <__main__.TensorPipeRpcTest testMethod=test_async_class_method>, <__main__.TensorPipeRpcTest testMethod=test_async_class_method_remote>, <__main__.TensorPipeRpcTest testMethod=test_async_class_rref_proxy>, <__main__.TensorPipeRpcTest testMethod=test_async_class_rref_proxy_async>, <__main__.TensorPipeRpcTest testMethod=test_async_class_rref_proxy_remote>, <__main__.TensorPipeRpcTest testMethod=test_async_function_chained>, <__main__.TensorPipeRpcTest testMethod=test_async_function_chained_remote>, <__main__.TensorPipeRpcTest testMethod=test_async_function_multi_chained>, <__main__.TensorPipeRpcTest testMethod=test_async_function_multi_chained_async>, <__main__.TensorPipeRpcTest testMethod=test_async_function_multi_chained_remote>, <__main__.TensorPipeRpcTest testMethod=test_async_function_multi_fanout>, <__main__.TensorPipeRpcTest testMethod=test_async_function_multi_fanout_async>, <__main__.TensorPipeRpcTest testMethod=test_async_function_multi_fanout_remote>, <__main__.TensorPipeRpcTest testMethod=test_async_function_nested>, <__main__.TensorPipeRpcTest testMethod=test_async_function_nested_remote>, <__main__.TensorPipeRpcTest testMethod=test_async_function_raise>, <__main__.TensorPipeRpcTest testMethod=test_async_function_raise_async>, <__main__.TensorPipeRpcTest testMethod=test_async_function_raise_remote>, <__main__.TensorPipeRpcTest testMethod=test_async_function_simple>, <__main__.TensorPipeRpcTest testMethod=test_async_function_with_future_ctor>, <__main__.TensorPipeRpcTest testMethod=test_async_function_with_future_ctor_remote>, <__main__.TensorPipeRpcTest testMethod=test_async_function_wrong_return_type>, <__main__.TensorPipeRpcTest testMethod=test_async_function_wrong_return_type_async>, <__main__.TensorPipeRpcTest testMethod=test_async_function_wrong_return_type_remote>, <__main__.TensorPipeRpcTest testMethod=test_async_record_function_cbs_jit_call>, <__main__.TensorPipeRpcTest testMethod=test_async_record_function_double_end_callbacks>, <__main__.TensorPipeRpcTest testMethod=test_async_record_function_double_end_callbacks_new_signatures>, <__main__.TensorPipeRpcTest testMethod=test_async_static_method>, <__main__.TensorPipeRpcTest testMethod=test_async_static_method_remote>, <__main__.TensorPipeRpcTest testMethod=test_build_rpc_profiling_key>, <__main__.TensorPipeRpcTest testMethod=test_builtin_remote_ret>, <__main__.TensorPipeRpcTest testMethod=test_builtin_remote_self>, <__main__.TensorPipeRpcTest testMethod=test_call_method_on_rref>, <__main__.TensorPipeRpcTest testMethod=test_callback_chain>, <__main__.TensorPipeRpcTest testMethod=test_callback_in_rpc>, <__main__.TensorPipeRpcTest testMethod=test_callback_multi>, <__main__.TensorPipeRpcTest testMethod=test_callback_none>, <__main__.TensorPipeRpcTest testMethod=test_callback_simple>, <__main__.TensorPipeRpcTest testMethod=test_callback_with_error>, <__main__.TensorPipeRpcTest testMethod=test_callback_with_ret>, <__main__.TensorPipeRpcTest testMethod=test_callback_wrong_arg_num>, <__main__.TensorPipeRpcTest testMethod=test_callback_wrong_arg_type>, <__main__.TensorPipeRpcTest testMethod=test_cannot_infer_backend_from_options>, <__main__.TensorPipeRpcTest testMethod=test_deadlock>, <__main__.TensorPipeRpcTest testMethod=test_debug_info>, <__main__.TensorPipeRpcTest testMethod=test_default_timeout_used>, <__main__.TensorPipeRpcTest testMethod=test_disable_gil_profiling>, <__main__.TensorPipeRpcTest testMethod=test_dist_init_decorator>, <__main__.TensorPipeRpcTest testMethod=test_duplicate_name>, <__main__.TensorPipeRpcTest testMethod=test_duplicate_name_2>, <__main__.TensorPipeRpcTest testMethod=test_expected_src>, <__main__.TensorPipeRpcTest testMethod=test_function_not_on_callee>, <__main__.TensorPipeRpcTest testMethod=test_future_done>, <__main__.TensorPipeRpcTest testMethod=test_future_done_exception>, <__main__.TensorPipeRpcTest testMethod=test_future_in_rpc>, <__main__.TensorPipeRpcTest testMethod=test_future_nested_callback>, <__main__.TensorPipeRpcTest testMethod=test_future_wait_twice>, <__main__.TensorPipeRpcTest testMethod=test_get_worker_infos>, <__main__.TensorPipeRpcTest testMethod=test_graceful_shutdown_with_uneven_workload>, <__main__.TensorPipeRpcTest testMethod=test_handle_send_exceptions>, <__main__.TensorPipeRpcTest testMethod=test_ignore_rref_leak>, <__main__.TensorPipeRpcTest testMethod=test_init_dynamic_and_static_rpc_group>, <__main__.TensorPipeRpcTest testMethod=test_init_pg_then_rpc>, <__main__.TensorPipeRpcTest testMethod=test_init_rpc_then_pg>, <__main__.TensorPipeRpcTest testMethod=test_init_rpc_twice>, <__main__.TensorPipeRpcTest testMethod=test_init_rpc_without_world_size>, <__main__.TensorPipeRpcTest testMethod=test_init_rpc_without_world_size_without_rank>, <__main__.TensorPipeRpcTest testMethod=test_int_callee>, <__main__.TensorPipeRpcTest testMethod=test_invalid_names>, <__main__.TensorPipeRpcTest testMethod=test_local_rref_no_fork>, <__main__.TensorPipeRpcTest testMethod=test_local_shutdown>, <__main__.TensorPipeRpcTest testMethod=test_local_shutdown_with_rpc>, <__main__.TensorPipeRpcTest testMethod=test_local_value_not_on_owner>, <__main__.TensorPipeRpcTest testMethod=test_mark_future_twice>, <__main__.TensorPipeRpcTest testMethod=test_multi_builtin_remote_ret>, <__main__.TensorPipeRpcTest testMethod=test_multi_layer_nested_async_rpc>, <__main__.TensorPipeRpcTest testMethod=test_multi_py_udf_remote>, <__main__.TensorPipeRpcTest testMethod=test_multi_rpc>, <__main__.TensorPipeRpcTest testMethod=test_my_parameter_server>, <__main__.TensorPipeRpcTest testMethod=test_nested_remote>, <__main__.TensorPipeRpcTest testMethod=test_nested_rpc>, <__main__.TensorPipeRpcTest testMethod=test_nested_rref>, <__main__.TensorPipeRpcTest testMethod=test_nested_rref_stress>, <__main__.TensorPipeRpcTest testMethod=test_non_cont_tensors>, <__main__.TensorPipeRpcTest testMethod=test_non_garbage_collected_user_rref_due_to_local_circular_dependency>, <__main__.TensorPipeRpcTest testMethod=test_nonzero>, <__main__.TensorPipeRpcTest testMethod=test_owner_equality>, <__main__.TensorPipeRpcTest testMethod=test_owner_rref_backward>, <__main__.TensorPipeRpcTest testMethod=test_pass_local_rrefs>, <__main__.TensorPipeRpcTest testMethod=test_pg_init_no_rpc_init>, <__main__.TensorPipeRpcTest testMethod=test_pickle_future>, <__main__.TensorPipeRpcTest testMethod=test_profiler_export_trace>, <__main__.TensorPipeRpcTest testMethod=test_profiler_remote_events_profiled>, <__main__.TensorPipeRpcTest testMethod=test_profiler_remote_events_profiled_single_threaded>, <__main__.TensorPipeRpcTest testMethod=test_profiler_rpc_key_names>, <__main__.TensorPipeRpcTest testMethod=test_profiler_rpc_memory>, <__main__.TensorPipeRpcTest testMethod=test_profiler_rpc_record_shapes>, <__main__.TensorPipeRpcTest testMethod=test_profiler_with_async_rpc_builtin>, <__main__.TensorPipeRpcTest testMethod=test_profiler_with_async_rpc_builtin_single_threaded>, <__main__.TensorPipeRpcTest testMethod=test_profiler_with_async_rpc_udf>, <__main__.TensorPipeRpcTest testMethod=test_profiler_with_async_rpc_udf_single_threaded>, <__main__.TensorPipeRpcTest testMethod=test_profiler_with_autograd_context>, <__main__.TensorPipeRpcTest testMethod=test_profiler_with_autograd_context_single_threaded>, <__main__.TensorPipeRpcTest testMethod=test_profiler_with_remote_builtin>, <__main__.TensorPipeRpcTest testMethod=test_profiler_with_remote_builtin_single_threaded>, <__main__.TensorPipeRpcTest testMethod=test_profiler_with_remote_udf>, <__main__.TensorPipeRpcTest testMethod=test_profiler_with_remote_udf_single_threaded>, <__main__.TensorPipeRpcTest testMethod=test_profiler_with_script_async_rpc>, <__main__.TensorPipeRpcTest testMethod=test_profiler_with_script_async_rpc_single_threaded>, <__main__.TensorPipeRpcTest testMethod=test_profiler_with_script_remote_rpc>, <__main__.TensorPipeRpcTest testMethod=test_profiler_with_script_remote_rpc_single_threaded>, <__main__.TensorPipeRpcTest testMethod=test_profiler_with_script_sync_rpc>, <__main__.TensorPipeRpcTest testMethod=test_profiler_with_script_sync_rpc_single_threaded>, <__main__.TensorPipeRpcTest testMethod=test_profiler_with_sync_rpc_builtin>, <__main__.TensorPipeRpcTest testMethod=test_profiler_with_sync_rpc_builtin_single_threaded>, <__main__.TensorPipeRpcTest testMethod=test_profiler_with_sync_rpc_udf>, <__main__.TensorPipeRpcTest testMethod=test_profiler_with_sync_rpc_udf_single_threaded>, <__main__.TensorPipeRpcTest testMethod=test_py_built_in>, <__main__.TensorPipeRpcTest testMethod=test_py_class_constructor>, <__main__.TensorPipeRpcTest testMethod=test_py_class_instance_method>, <__main__.TensorPipeRpcTest testMethod=test_py_class_method>, <__main__.TensorPipeRpcTest testMethod=test_py_class_static_method>, <__main__.TensorPipeRpcTest testMethod=test_py_function_exception>, <__main__.TensorPipeRpcTest testMethod=test_py_multi_async_call>, <__main__.TensorPipeRpcTest testMethod=test_py_nested_pickle>, <__main__.TensorPipeRpcTest testMethod=test_py_no_return_result>, <__main__.TensorPipeRpcTest testMethod=test_py_raise_in_user_func>, <__main__.TensorPipeRpcTest testMethod=test_py_raise_in_user_func_escaped_str>, <__main__.TensorPipeRpcTest testMethod=test_py_rpc_rref_args>, <__main__.TensorPipeRpcTest testMethod=test_py_rref_args>, <__main__.TensorPipeRpcTest testMethod=test_py_rref_args_user_share>, <__main__.TensorPipeRpcTest testMethod=test_py_tensors>, <__main__.TensorPipeRpcTest testMethod=test_py_tensors_in_container>, <__main__.TensorPipeRpcTest testMethod=test_py_tensors_multi_async_call>, <__main__.TensorPipeRpcTest testMethod=test_py_udf_remote>, <__main__.TensorPipeRpcTest testMethod=test_py_user_defined>, <__main__.TensorPipeRpcTest testMethod=test_register_rpc_backend_and_set_and_start_rpc_backend>, <__main__.TensorPipeRpcTest testMethod=test_reinit>, <__main__.TensorPipeRpcTest testMethod=test_remote_same_worker>, <__main__.TensorPipeRpcTest testMethod=test_remote_throw>, <__main__.TensorPipeRpcTest testMethod=test_remote_with_exception>, <__main__.TensorPipeRpcTest testMethod=test_return_future>, <__main__.TensorPipeRpcTest testMethod=test_return_future_async>, <__main__.TensorPipeRpcTest testMethod=test_return_future_remote>, <__main__.TensorPipeRpcTest testMethod=test_return_local_rrefs>, <__main__.TensorPipeRpcTest testMethod=test_rpc_barrier_all>, <__main__.TensorPipeRpcTest testMethod=test_rpc_barrier_multithreaded>, <__main__.TensorPipeRpcTest testMethod=test_rpc_barrier_partial_subset>, <__main__.TensorPipeRpcTest testMethod=test_rpc_barrier_subset>, <__main__.TensorPipeRpcTest testMethod=test_rpc_profiling_async_function>, <__main__.TensorPipeRpcTest testMethod=test_rpc_profiling_async_function_single_threaded>, <__main__.TensorPipeRpcTest testMethod=test_rpc_profiling_remote_record_function>, <__main__.TensorPipeRpcTest testMethod=test_rpc_return_rref>, <__main__.TensorPipeRpcTest testMethod=test_rpc_timeouts>, <__main__.TensorPipeRpcTest testMethod=test_rref_context_debug_info>, <__main__.TensorPipeRpcTest testMethod=test_rref_forward_chain>, <__main__.TensorPipeRpcTest testMethod=test_rref_get_future>, <__main__.TensorPipeRpcTest testMethod=test_rref_leak>, <__main__.TensorPipeRpcTest testMethod=test_rref_proxy_class>, <__main__.TensorPipeRpcTest testMethod=test_rref_proxy_class_self>, <__main__.TensorPipeRpcTest testMethod=test_rref_proxy_non_exist>, <__main__.TensorPipeRpcTest testMethod=test_rref_proxy_reuse>, <__main__.TensorPipeRpcTest testMethod=test_rref_proxy_tensor>, <__main__.TensorPipeRpcTest testMethod=test_rref_proxy_tensor_self>, <__main__.TensorPipeRpcTest testMethod=test_rref_py_pickle_not_supported>, <__main__.TensorPipeRpcTest testMethod=test_rref_str>, <__main__.TensorPipeRpcTest testMethod=test_rref_timeout>, <__main__.TensorPipeRpcTest testMethod=test_rref_type_blocking>, <__main__.TensorPipeRpcTest testMethod=test_rref_type_non_blocking>, <__main__.TensorPipeRpcTest testMethod=test_rref_type_owner_blocking>, <__main__.TensorPipeRpcTest testMethod=test_rref_type_owner_non_blocking>, <__main__.TensorPipeRpcTest testMethod=test_rref_type_slow_init>, <__main__.TensorPipeRpcTest testMethod=test_rref_type_with_error_blocking>, <__main__.TensorPipeRpcTest testMethod=test_rref_type_with_error_non_blocking>, <__main__.TensorPipeRpcTest testMethod=test_scalar_add>, <__main__.TensorPipeRpcTest testMethod=test_self_add>, <__main__.TensorPipeRpcTest testMethod=test_self_py_udf_remote>, <__main__.TensorPipeRpcTest testMethod=test_self_remote_rref_as_remote_arg>, <__main__.TensorPipeRpcTest testMethod=test_self_remote_rref_as_rpc_arg>, <__main__.TensorPipeRpcTest testMethod=test_self_remote_rref_as_self_remote_arg>, <__main__.TensorPipeRpcTest testMethod=test_self_remote_rref_as_self_rpc_arg>, <__main__.TensorPipeRpcTest testMethod=test_send_to_rank>, <__main__.TensorPipeRpcTest testMethod=test_server_process_global_profiler>, <__main__.TensorPipeRpcTest testMethod=test_set_and_get_default_rpc_timeout>, <__main__.TensorPipeRpcTest testMethod=test_shutdown_errors>, <__main__.TensorPipeRpcTest testMethod=test_shutdown_followed_by_rpc>, <__main__.TensorPipeRpcTest testMethod=test_stress_heavy_rpc>, <__main__.TensorPipeRpcTest testMethod=test_stress_heavy_rpc_torchscript>, <__main__.TensorPipeRpcTest testMethod=test_stress_light_rpc>, <__main__.TensorPipeRpcTest testMethod=test_use_rpc_pickler>, <__main__.TensorPipeRpcTest testMethod=test_use_rref_after_shutdown>, <__main__.TensorPipeRpcTest testMethod=test_user_rref_backward>, <__main__.TensorPipeRpcTest testMethod=test_user_rrefs_confirmed>, <__main__.TensorPipeRpcTest testMethod=test_user_rrefs_confirmed_remote>, <__main__.TensorPipeRpcTest testMethod=test_wait_all>, <__main__.TensorPipeRpcTest testMethod=test_wait_all_exit_early_builtin>, <__main__.TensorPipeRpcTest testMethod=test_wait_all_exit_early_python>, <__main__.TensorPipeRpcTest testMethod=test_wait_all_exit_early_script_function>, <__main__.TensorPipeRpcTest testMethod=test_wait_all_multiple_call>, <__main__.TensorPipeRpcTest testMethod=test_wait_all_raise_in_body>, <__main__.TensorPipeRpcTest testMethod=test_wait_all_raise_in_user_func>, <__main__.TensorPipeRpcTest testMethod=test_wait_all_timeout>, <__main__.TensorPipeRpcTest testMethod=test_wait_all_with_exception>, <__main__.TensorPipeRpcTest testMethod=test_wait_all_with_partial_exception>, <__main__.TensorPipeRpcTest testMethod=test_wait_all_workers_dense>, <__main__.TensorPipeRpcTest testMethod=test_wait_all_workers_timeout>, <__main__.TensorPipeRpcTest testMethod=test_wait_all_workers_twice_dense>, <__main__.TensorPipeRpcTest testMethod=test_without_world_size_existing_rank_can_communicate_with_new_rank>, <__main__.TensorPipeRpcTest testMethod=test_without_world_size_existing_rank_can_communicate_with_new_rank_cuda>, <__main__.TensorPipeRpcTest testMethod=test_without_world_size_new_rank_can_communicated_with_existing_rank>, <__main__.TensorPipeRpcTest testMethod=test_worker_id>, <__main__.TensorPipeRpcTest testMethod=test_worker_info_pickle>, <__main__.TensorPipeRpcTest testMethod=test_world_size_one>, <__main__.TensorPipeRpcTest testMethod=test_wrong_types>]>
2022-05-18T03:51:40.0636067Z test_add (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0636314Z test_add_done_callback (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0636573Z test_add_with_id (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0636817Z test_all_gather (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0637058Z test_all_gather_timeout (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0637310Z test_async_add (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0637562Z test_async_class_method (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0637817Z test_async_class_method_remote (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0638090Z test_async_class_rref_proxy (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0638368Z test_async_class_rref_proxy_async (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0638652Z test_async_class_rref_proxy_remote (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0638915Z test_async_function_chained (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0639198Z test_async_function_chained_remote (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0639488Z test_async_function_multi_chained (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0639766Z test_async_function_multi_chained_async (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0640067Z test_async_function_multi_chained_remote (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0640356Z test_async_function_multi_fanout (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0640627Z test_async_function_multi_fanout_async (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0640921Z test_async_function_multi_fanout_remote (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0641204Z test_async_function_nested (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0641481Z test_async_function_nested_remote (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0641740Z test_async_function_raise (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0642017Z test_async_function_raise_async (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0642298Z test_async_function_raise_remote (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0642557Z test_async_function_simple (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0642837Z test_async_function_with_future_ctor (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0643137Z test_async_function_with_future_ctor_remote (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0643510Z test_async_function_wrong_return_type (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0643810Z test_async_function_wrong_return_type_async (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0644116Z test_async_function_wrong_return_type_remote (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0644422Z test_async_record_function_cbs_jit_call (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0644716Z test_async_record_function_double_end_callbacks (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0645051Z test_async_record_function_double_end_callbacks_new_signatures (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0645439Z test_async_static_method (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0645697Z test_async_static_method_remote (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0646015Z test_build_rpc_profiling_key (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0646283Z test_builtin_remote_ret (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0646532Z test_builtin_remote_self (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0646790Z test_call_method_on_rref (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0647047Z test_callback_chain (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0647299Z test_callback_in_rpc (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0647539Z test_callback_multi (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0647790Z test_callback_none (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0648040Z test_callback_simple (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0648284Z test_callback_with_error (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0648548Z test_callback_with_ret (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0648812Z test_callback_wrong_arg_num (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0649070Z test_callback_wrong_arg_type (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0649355Z test_cannot_infer_backend_from_options (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0649624Z test_deadlock (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0649864Z test_debug_info (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0650107Z test_default_timeout_used (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0650375Z test_disable_gil_profiling (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0650644Z test_dist_init_decorator (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0650890Z test_duplicate_name (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0651142Z test_duplicate_name_2 (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0651394Z test_expected_src (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0651640Z test_function_not_on_callee (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0651899Z test_future_done (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0652156Z test_future_done_exception (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0652401Z test_future_in_rpc (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0652662Z test_future_nested_callback (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0652925Z test_future_wait_twice (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0653180Z test_get_worker_infos (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0653450Z test_graceful_shutdown_with_uneven_workload (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0653741Z test_handle_send_exceptions (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0654002Z test_ignore_rref_leak (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0654267Z test_init_dynamic_and_static_rpc_group (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0654540Z test_init_pg_then_rpc (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0654794Z test_init_rpc_then_pg (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0655033Z test_init_rpc_twice (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0655299Z test_init_rpc_without_world_size (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0655596Z test_init_rpc_without_world_size_without_rank (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0655859Z test_int_callee (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0656106Z test_invalid_names (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0656361Z test_local_rref_no_fork (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0656615Z test_local_shutdown (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0656863Z test_local_shutdown_with_rpc (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0657133Z test_local_value_not_on_owner (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0657394Z test_mark_future_twice (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0657644Z test_multi_builtin_remote_ret (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0657923Z test_multi_layer_nested_async_rpc (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0658230Z test_multi_py_udf_remote (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0658465Z test_multi_rpc (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0658932Z test_my_parameter_server (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0659198Z test_nested_remote (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0659429Z test_nested_rpc (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0659674Z test_nested_rref (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0659925Z test_nested_rref_stress (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0660182Z test_non_cont_tensors (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0660486Z test_non_garbage_collected_user_rref_due_to_local_circular_dependency (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0660786Z test_nonzero (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0661030Z test_owner_equality (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0661275Z test_owner_rref_backward (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0661531Z test_pass_local_rrefs (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0661793Z test_pg_init_no_rpc_init (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0662035Z test_pickle_future (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0662296Z test_profiler_export_trace (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0662577Z test_profiler_remote_events_profiled (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0663068Z test_profiler_remote_events_profiled_single_threaded (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0663365Z test_profiler_rpc_key_names (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0663633Z test_profiler_rpc_memory (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0663906Z test_profiler_rpc_record_shapes (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0664177Z test_profiler_with_async_rpc_builtin (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0664488Z test_profiler_with_async_rpc_builtin_single_threaded (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0664792Z test_profiler_with_async_rpc_udf (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0665084Z test_profiler_with_async_rpc_udf_single_threaded (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0665392Z test_profiler_with_autograd_context (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0665705Z test_profiler_with_autograd_context_single_threaded (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0666013Z test_profiler_with_remote_builtin (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0666304Z test_profiler_with_remote_builtin_single_threaded (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0666606Z test_profiler_with_remote_udf (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0666899Z test_profiler_with_remote_udf_single_threaded (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0667183Z test_profiler_with_script_async_rpc (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0667491Z test_profiler_with_script_async_rpc_single_threaded (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0667798Z test_profiler_with_script_remote_rpc (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0668168Z test_profiler_with_script_remote_rpc_single_threaded (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0668463Z test_profiler_with_script_sync_rpc (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0668772Z test_profiler_with_script_sync_rpc_single_threaded (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0669076Z test_profiler_with_sync_rpc_builtin (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0669447Z test_profiler_with_sync_rpc_builtin_single_threaded (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0669783Z test_profiler_with_sync_rpc_udf (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0670082Z test_profiler_with_sync_rpc_udf_single_threaded (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0670345Z test_py_built_in (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0670601Z test_py_class_constructor (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0670875Z test_py_class_instance_method (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0671136Z test_py_class_method (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0671469Z test_py_class_static_method (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0671737Z test_py_function_exception (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0672044Z test_py_multi_async_call (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0672290Z test_py_nested_pickle (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0672549Z test_py_no_return_result (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0672811Z test_py_raise_in_user_func (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0673075Z test_py_raise_in_user_func_escaped_str (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0673349Z test_py_rpc_rref_args (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0673601Z test_py_rref_args (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0673858Z test_py_rref_args_user_share (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0674100Z test_py_tensors (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0674359Z test_py_tensors_in_container (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0674634Z test_py_tensors_multi_async_call (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0674884Z test_py_udf_remote (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0675138Z test_py_user_defined (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0675436Z test_register_rpc_backend_and_set_and_start_rpc_backend (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0675705Z test_reinit (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0675954Z test_remote_same_worker (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0676208Z test_remote_throw (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0676452Z test_remote_with_exception (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0676712Z test_return_future (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0676967Z test_return_future_async (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0677227Z test_return_future_remote (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0677475Z test_return_local_rrefs (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0677732Z test_rpc_barrier_all (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0678003Z test_rpc_barrier_multithreaded (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0678276Z test_rpc_barrier_partial_subset (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0678545Z test_rpc_barrier_subset (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0678816Z test_rpc_profiling_async_function (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0679107Z test_rpc_profiling_async_function_single_threaded (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0679423Z test_rpc_profiling_remote_record_function (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0679701Z test_rpc_return_rref (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0679952Z test_rpc_timeouts (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0680198Z test_rref_context_debug_info (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0680464Z test_rref_forward_chain (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0680718Z test_rref_get_future (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0680952Z test_rref_leak (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0681198Z test_rref_proxy_class (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0681463Z test_rref_proxy_class_self (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0681712Z test_rref_proxy_non_exist (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0681969Z test_rref_proxy_reuse (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0682226Z test_rref_proxy_tensor (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0682476Z test_rref_proxy_tensor_self (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0682753Z test_rref_py_pickle_not_supported (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0683015Z test_rref_str (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0683258Z test_rref_timeout (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0683497Z test_rref_type_blocking (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0683762Z test_rref_type_non_blocking (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0684032Z test_rref_type_owner_blocking (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0684334Z test_rref_type_owner_non_blocking (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0684607Z test_rref_type_slow_init (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0684918Z test_rref_type_with_error_blocking (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0685194Z test_rref_type_with_error_non_blocking (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0685462Z test_scalar_add (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0685707Z test_self_add (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0685943Z test_self_py_udf_remote (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0686213Z test_self_remote_rref_as_remote_arg (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0686493Z test_self_remote_rref_as_rpc_arg (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0686779Z test_self_remote_rref_as_self_remote_arg (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0687054Z test_self_remote_rref_as_self_rpc_arg (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0687324Z test_send_to_rank (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0687591Z test_server_process_global_profiler (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0687868Z test_set_and_get_default_rpc_timeout (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0688141Z test_shutdown_errors (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0688410Z test_shutdown_followed_by_rpc (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0688659Z test_stress_heavy_rpc (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0688931Z test_stress_heavy_rpc_torchscript (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0689204Z test_stress_light_rpc (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0689459Z test_use_rpc_pickler (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0689713Z test_use_rref_after_shutdown (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0689978Z test_user_rref_backward (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0690241Z test_user_rrefs_confirmed (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0690503Z test_user_rrefs_confirmed_remote (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0690759Z test_wait_all (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0691021Z test_wait_all_exit_early_builtin (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0691283Z test_wait_all_exit_early_python (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0691569Z test_wait_all_exit_early_script_function (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0691851Z test_wait_all_multiple_call (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0692117Z test_wait_all_raise_in_body (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0692375Z test_wait_all_raise_in_user_func (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0692635Z test_wait_all_timeout (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0692897Z test_wait_all_with_exception (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0693162Z test_wait_all_with_partial_exception (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0693441Z test_wait_all_workers_dense (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0693716Z test_wait_all_workers_timeout (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0693978Z test_wait_all_workers_twice_dense (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0694299Z test_without_world_size_existing_rank_can_communicate_with_new_rank (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0694664Z test_without_world_size_existing_rank_can_communicate_with_new_rank_cuda (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0695025Z test_without_world_size_new_rank_can_communicated_with_existing_rank (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0695309Z test_worker_id (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0695561Z test_worker_info_pickle (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0695818Z test_world_size_one (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0696052Z test_wrong_types (__main__.TensorPipeRpcTest)
2022-05-18T03:51:40.0700730Z <unittest.suite.TestSuite tests=[<__main__.TensorPipeTensorPipeAgentDistAutogradTest testMethod=test_backward_different_dtypes_sparse>, <__main__.TensorPipeTensorPipeAgentDistAutogradTest testMethod=test_backward_multiple_round_trips_sparse>, <__main__.TensorPipeTensorPipeAgentDistAutogradTest testMethod=test_backward_no_grad_on_tensor_sparse>, <__main__.TensorPipeTensorPipeAgentDistAutogradTest testMethod=test_backward_rref_multi_sparse>, <__main__.TensorPipeTensorPipeAgentDistAutogradTest testMethod=test_backward_rref_nested_sparse>, <__main__.TensorPipeTensorPipeAgentDistAutogradTest testMethod=test_backward_rref_sparse>, <__main__.TensorPipeTensorPipeAgentDistAutogradTest testMethod=test_backward_simple_python_udf_sparse>, <__main__.TensorPipeTensorPipeAgentDistAutogradTest testMethod=test_backward_simple_script_call_sparse>, <__main__.TensorPipeTensorPipeAgentDistAutogradTest testMethod=test_backward_simple_self_sparse>, <__main__.TensorPipeTensorPipeAgentDistAutogradTest testMethod=test_backward_simple_sparse>, <__main__.TensorPipeTensorPipeAgentDistAutogradTest testMethod=test_backwards_nested_python_udf_sparse>, <__main__.TensorPipeTensorPipeAgentDistAutogradTest testMethod=test_context_cleanup_nested_rpc_sparse>, <__main__.TensorPipeTensorPipeAgentDistAutogradTest testMethod=test_context_cleanup_tensor_no_grad_sparse>, <__main__.TensorPipeTensorPipeAgentDistAutogradTest testMethod=test_context_cleanup_tensor_with_grad_sparse>, <__main__.TensorPipeTensorPipeAgentDistAutogradTest testMethod=test_embedding_bag_with_no_grad_tensors>, <__main__.TensorPipeTensorPipeAgentDistAutogradTest testMethod=test_graph_for_builtin_call_sparse>, <__main__.TensorPipeTensorPipeAgentDistAutogradTest testMethod=test_graph_for_builtin_remote_call_sparse>, <__main__.TensorPipeTensorPipeAgentDistAutogradTest testMethod=test_graph_for_py_nested_call_itself_sparse>, <__main__.TensorPipeTensorPipeAgentDistAutogradTest testMethod=test_graph_for_py_nested_call_sparse>, <__main__.TensorPipeTensorPipeAgentDistAutogradTest testMethod=test_graph_for_py_nested_remote_call_itself_sparse>, <__main__.TensorPipeTensorPipeAgentDistAutogradTest testMethod=test_graph_for_py_nested_remote_call_sparse>, <__main__.TensorPipeTensorPipeAgentDistAutogradTest testMethod=test_graph_for_python_call_sparse>, <__main__.TensorPipeTensorPipeAgentDistAutogradTest testMethod=test_graph_for_python_remote_call_sparse>, <__main__.TensorPipeTensorPipeAgentDistAutogradTest testMethod=test_mixed_requires_grad_sparse>, <__main__.TensorPipeTensorPipeAgentDistAutogradTest testMethod=test_multiple_backward_sparse>, <__main__.TensorPipeTensorPipeAgentDistAutogradTest testMethod=test_nested_backward_accumulate_grads_sparse>, <__main__.TensorPipeTensorPipeAgentDistAutogradTest testMethod=test_no_graph_with_tensors_not_require_grad_remote_sparse>, <__main__.TensorPipeTensorPipeAgentDistAutogradTest testMethod=test_no_graph_with_tensors_not_require_grad_sparse>, <__main__.TensorPipeTensorPipeAgentDistAutogradTest testMethod=test_remote_complex_args_sparse>, <__main__.TensorPipeTensorPipeAgentDistAutogradTest testMethod=test_rpc_complex_args_sparse>, <__main__.TensorPipeTensorPipeAgentDistAutogradTest testMethod=test_trainer_ps_sparse>]>
2022-05-18T03:51:40.0705499Z test_backward_different_dtypes_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest)
2022-05-18T03:51:40.0705925Z test_backward_multiple_round_trips_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest)
2022-05-18T03:51:40.0706338Z test_backward_no_grad_on_tensor_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest)
2022-05-18T03:51:40.0706741Z test_backward_rref_multi_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest)
2022-05-18T03:51:40.0707121Z test_backward_rref_nested_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest)
2022-05-18T03:51:40.0707509Z test_backward_rref_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest)
2022-05-18T03:51:40.0707904Z test_backward_simple_python_udf_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest)
2022-05-18T03:51:40.0708299Z test_backward_simple_script_call_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest)
2022-05-18T03:51:40.0708701Z test_backward_simple_self_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest)
2022-05-18T03:51:40.0709153Z test_backward_simple_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest)
2022-05-18T03:51:40.0709583Z test_backwards_nested_python_udf_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest)
2022-05-18T03:51:40.0709977Z test_context_cleanup_nested_rpc_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest)
2022-05-18T03:51:40.0710390Z test_context_cleanup_tensor_no_grad_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest)
2022-05-18T03:51:40.0710807Z test_context_cleanup_tensor_with_grad_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest)
2022-05-18T03:51:40.0711207Z test_embedding_bag_with_no_grad_tensors (__main__.TensorPipeTensorPipeAgentDistAutogradTest)
2022-05-18T03:51:40.0711608Z test_graph_for_builtin_call_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest)
2022-05-18T03:51:40.0712012Z test_graph_for_builtin_remote_call_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest)
2022-05-18T03:51:40.0712425Z test_graph_for_py_nested_call_itself_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest)
2022-05-18T03:51:40.0712818Z test_graph_for_py_nested_call_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest)
2022-05-18T03:51:40.0713234Z test_graph_for_py_nested_remote_call_itself_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest)
2022-05-18T03:51:40.0713653Z test_graph_for_py_nested_remote_call_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest)
2022-05-18T03:51:40.0714051Z test_graph_for_python_call_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest)
2022-05-18T03:51:40.0714441Z test_graph_for_python_remote_call_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest)
2022-05-18T03:51:40.0714837Z test_mixed_requires_grad_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest)
2022-05-18T03:51:40.0715226Z test_multiple_backward_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest)
2022-05-18T03:51:40.0715617Z test_nested_backward_accumulate_grads_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest)
2022-05-18T03:51:40.0716060Z test_no_graph_with_tensors_not_require_grad_remote_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest)
2022-05-18T03:51:40.0716496Z test_no_graph_with_tensors_not_require_grad_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest)
2022-05-18T03:51:40.0716900Z test_remote_complex_args_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest)
2022-05-18T03:51:40.0717275Z test_rpc_complex_args_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest)
2022-05-18T03:51:40.0717652Z test_trainer_ps_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest)
2022-05-18T03:51:40.0721928Z <unittest.suite.TestSuite tests=[<__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_builtin_remote_ret_sparse>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_builtin_remote_self_sparse>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_infer_backend_from_options>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_meta_multiple_tensors>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_meta_one_tensor>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_meta_one_tensor_rref>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_mismatched_type_for_options>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_multi_builtin_remote_ret_sparse>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_multi_py_udf_remote_sparse>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_multi_rpc_sparse>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_my_parameter_server_sparse>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_nested_remote_sparse>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_nested_rpc_sparse>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_nested_rref_sparse>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_nested_rref_stress_sparse>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_op_with_invalid_args>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_py_rpc_rref_args_sparse>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_py_rref_args_sparse>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_py_rref_args_user_share_sparse>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_py_sparse_tensors_in_container>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_rref_get_type_timeout_blocking>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_rref_get_type_timeout_non_blocking>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_rref_proxy_timeout>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_self_py_udf_remote_sparse>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_self_remote_rref_as_remote_arg_sparse>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_self_remote_rref_as_rpc_arg_sparse>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_self_remote_rref_as_self_remote_arg_sparse>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_self_remote_rref_as_self_rpc_arg_sparse>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_send_to_rank_sparse>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_set_and_get_num_worker_threads>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_stress_heavy_rpc_sparse>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_tensorpipe_options_throw_on_timedelta_timeout>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_tensorpipe_set_default_timeout>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_wait_all_workers_sparse>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_wait_all_workers_twice_sparse>, <__main__.TensorPipeTensorPipeAgentRpcTest testMethod=test_world_size_one_sparse>]>
2022-05-18T03:51:40.0726091Z test_builtin_remote_ret_sparse (__main__.TensorPipeTensorPipeAgentRpcTest)
2022-05-18T03:51:40.0726440Z test_builtin_remote_self_sparse (__main__.TensorPipeTensorPipeAgentRpcTest)
2022-05-18T03:51:40.0726776Z test_infer_backend_from_options (__main__.TensorPipeTensorPipeAgentRpcTest)
2022-05-18T03:51:40.0727115Z test_meta_multiple_tensors (__main__.TensorPipeTensorPipeAgentRpcTest)
2022-05-18T03:51:40.0727443Z test_meta_one_tensor (__main__.TensorPipeTensorPipeAgentRpcTest)
2022-05-18T03:51:40.0727768Z test_meta_one_tensor_rref (__main__.TensorPipeTensorPipeAgentRpcTest)
2022-05-18T03:51:40.0728099Z test_mismatched_type_for_options (__main__.TensorPipeTensorPipeAgentRpcTest)
2022-05-18T03:51:40.0728453Z test_multi_builtin_remote_ret_sparse (__main__.TensorPipeTensorPipeAgentRpcTest)
2022-05-18T03:51:40.0728799Z test_multi_py_udf_remote_sparse (__main__.TensorPipeTensorPipeAgentRpcTest)
2022-05-18T03:51:40.0729119Z test_multi_rpc_sparse (__main__.TensorPipeTensorPipeAgentRpcTest)
2022-05-18T03:51:40.0729453Z test_my_parameter_server_sparse (__main__.TensorPipeTensorPipeAgentRpcTest)
2022-05-18T03:51:40.0729790Z test_nested_remote_sparse (__main__.TensorPipeTensorPipeAgentRpcTest)
2022-05-18T03:51:40.0730118Z test_nested_rpc_sparse (__main__.TensorPipeTensorPipeAgentRpcTest)
2022-05-18T03:51:40.0730434Z test_nested_rref_sparse (__main__.TensorPipeTensorPipeAgentRpcTest)
2022-05-18T03:51:40.0730765Z test_nested_rref_stress_sparse (__main__.TensorPipeTensorPipeAgentRpcTest)
2022-05-18T03:51:40.0731099Z test_op_with_invalid_args (__main__.TensorPipeTensorPipeAgentRpcTest)
2022-05-18T03:51:40.0731419Z test_py_rpc_rref_args_sparse (__main__.TensorPipeTensorPipeAgentRpcTest)
2022-05-18T03:51:40.0731751Z test_py_rref_args_sparse (__main__.TensorPipeTensorPipeAgentRpcTest)
2022-05-18T03:51:40.0732088Z test_py_rref_args_user_share_sparse (__main__.TensorPipeTensorPipeAgentRpcTest)
2022-05-18T03:51:40.0732437Z test_py_sparse_tensors_in_container (__main__.TensorPipeTensorPipeAgentRpcTest)
2022-05-18T03:51:40.0732774Z test_rref_get_type_timeout_blocking (__main__.TensorPipeTensorPipeAgentRpcTest)
2022-05-18T03:51:40.0733128Z test_rref_get_type_timeout_non_blocking (__main__.TensorPipeTensorPipeAgentRpcTest)
2022-05-18T03:51:40.0733502Z test_rref_proxy_timeout (__main__.TensorPipeTensorPipeAgentRpcTest)
2022-05-18T03:51:40.0733852Z test_self_py_udf_remote_sparse (__main__.TensorPipeTensorPipeAgentRpcTest)
2022-05-18T03:51:40.0734209Z test_self_remote_rref_as_remote_arg_sparse (__main__.TensorPipeTensorPipeAgentRpcTest)
2022-05-18T03:51:40.0734574Z test_self_remote_rref_as_rpc_arg_sparse (__main__.TensorPipeTensorPipeAgentRpcTest)
2022-05-18T03:51:40.0734942Z test_self_remote_rref_as_self_remote_arg_sparse (__main__.TensorPipeTensorPipeAgentRpcTest)
2022-05-18T03:51:40.0735295Z test_self_remote_rref_as_self_rpc_arg_sparse (__main__.TensorPipeTensorPipeAgentRpcTest)
2022-05-18T03:51:40.0735641Z test_send_to_rank_sparse (__main__.TensorPipeTensorPipeAgentRpcTest)
2022-05-18T03:51:40.0735983Z test_set_and_get_num_worker_threads (__main__.TensorPipeTensorPipeAgentRpcTest)
2022-05-18T03:51:40.0736313Z test_stress_heavy_rpc_sparse (__main__.TensorPipeTensorPipeAgentRpcTest)
2022-05-18T03:51:40.0736680Z test_tensorpipe_options_throw_on_timedelta_timeout (__main__.TensorPipeTensorPipeAgentRpcTest)
2022-05-18T03:51:40.0737053Z test_tensorpipe_set_default_timeout (__main__.TensorPipeTensorPipeAgentRpcTest)
2022-05-18T03:51:40.0737382Z test_wait_all_workers_sparse (__main__.TensorPipeTensorPipeAgentRpcTest)
2022-05-18T03:51:40.0737723Z test_wait_all_workers_twice_sparse (__main__.TensorPipeTensorPipeAgentRpcTest)
2022-05-18T03:51:40.0738061Z test_world_size_one_sparse (__main__.TensorPipeTensorPipeAgentRpcTest)
2022-05-18T03:51:40.0738813Z <unittest.suite.TestSuite tests=[<__main__.TensorPipeThreeWorkersRemoteModuleTest testMethod=test_create_remote_module_from_module_rref>, <__main__.TensorPipeThreeWorkersRemoteModuleTest testMethod=test_send_remote_module_over_the_wire>, <__main__.TensorPipeThreeWorkersRemoteModuleTest testMethod=test_send_remote_module_over_the_wire_script_not_supported>]>
2022-05-18T03:51:40.0739527Z test_create_remote_module_from_module_rref (__main__.TensorPipeThreeWorkersRemoteModuleTest)
2022-05-18T03:51:40.0739904Z test_send_remote_module_over_the_wire (__main__.TensorPipeThreeWorkersRemoteModuleTest)
2022-05-18T03:51:40.0740314Z test_send_remote_module_over_the_wire_script_not_supported (__main__.TensorPipeThreeWorkersRemoteModuleTest)
2022-05-18T03:51:40.6151087Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsuv7k0x9
2022-05-18T03:51:40.6152075Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsuv7k0x9/_remote_module_non_scriptable.py
2022-05-18T03:51:40.8685113Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T03:51:40.8695104Z 
2022-05-18T03:51:40.8695250Z Running tests...
2022-05-18T03:51:40.8695629Z ----------------------------------------------------------------------
2022-05-18T03:51:41.1782352Z   test_ddp_comparison (__main__.TensorPipeDdpComparisonTest) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/69662 for allplatform(s) . If you're seeing this on your local machine and would like to enable this test, please make sure IN_CI is not set and you are not using the flag --import-disabled-tests. (0.308s)
2022-05-18T03:51:41.1782873Z 
2022-05-18T03:51:41.1783297Z ----------------------------------------------------------------------
2022-05-18T03:51:41.1783541Z Ran 1 test in 0.309s
2022-05-18T03:51:41.1783654Z 
2022-05-18T03:51:41.1784164Z OK (skipped=1)
2022-05-18T03:51:41.1784306Z 
2022-05-18T03:51:41.1784398Z Generating XML reports...
2022-05-18T03:51:41.1806637Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDdpComparisonTest-20220518035140.xml
2022-05-18T03:51:41.8894151Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpent0bu_7
2022-05-18T03:51:41.8894917Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpent0bu_7/_remote_module_non_scriptable.py
2022-05-18T03:51:42.1426078Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T03:51:42.1435259Z 
2022-05-18T03:51:42.1435408Z Running tests...
2022-05-18T03:51:42.1435930Z ----------------------------------------------------------------------
2022-05-18T03:51:42.4525864Z   test_ddp_comparison_uneven_inputs (__main__.TensorPipeDdpComparisonTest) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/72891 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure IN_CI is not set and you are not using the flag --import-disabled-tests. (0.309s)
2022-05-18T03:51:42.4526813Z 
2022-05-18T03:51:42.4527023Z ----------------------------------------------------------------------
2022-05-18T03:51:42.4527284Z Ran 1 test in 0.309s
2022-05-18T03:51:42.4527397Z 
2022-05-18T03:51:42.4527457Z OK (skipped=1)
2022-05-18T03:51:42.4527566Z 
2022-05-18T03:51:42.4527651Z Generating XML reports...
2022-05-18T03:51:42.4549714Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDdpComparisonTest-20220518035142.xml
2022-05-18T03:51:43.1636628Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzjx04qbx
2022-05-18T03:51:43.1637273Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzjx04qbx/_remote_module_non_scriptable.py
2022-05-18T03:51:43.4176571Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T03:51:43.4186549Z 
2022-05-18T03:51:43.4186907Z Running tests...
2022-05-18T03:51:43.4187340Z ----------------------------------------------------------------------
2022-05-18T03:51:43.7323268Z   test_ddp_dist_autograd_local_vs_remote (__main__.TensorPipeDdpComparisonTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7043
2022-05-18T03:51:43.7345490Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7044
2022-05-18T03:51:43.7368266Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7045
2022-05-18T03:51:43.7392744Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7046
2022-05-18T03:51:44.3450469Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdp2rgzjt
2022-05-18T03:51:44.3451241Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdp2rgzjt/_remote_module_non_scriptable.py
2022-05-18T03:51:44.3570513Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwsjr884b
2022-05-18T03:51:44.3571241Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwsjr884b/_remote_module_non_scriptable.py
2022-05-18T03:51:44.3841637Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwqkdvw58
2022-05-18T03:51:44.3842679Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwqkdvw58/_remote_module_non_scriptable.py
2022-05-18T03:51:44.3887023Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8odowiyb
2022-05-18T03:51:44.3888580Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8odowiyb/_remote_module_non_scriptable.py
2022-05-18T03:51:44.5941630Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:51:44.6040503Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:51:44.6323302Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:51:44.6377188Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:51:44.8945318Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:51:44.9044385Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:51:44.9045413Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:51:44.9046465Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:51:44.9048615Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:51:44.9049907Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:51:44.9051068Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:51:44.9052368Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:51:44.9525802Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T03:51:44.9533007Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T03:51:44.9533586Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T03:51:44.9534009Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T03:51:44.9755078Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T03:51:44.9759165Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T03:51:44.9759887Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T03:51:44.9763962Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T03:51:45.2434047Z ok (1.824s)
2022-05-18T03:51:45.2434273Z 
2022-05-18T03:51:45.2434705Z ----------------------------------------------------------------------
2022-05-18T03:51:45.2435010Z Ran 1 test in 1.825s
2022-05-18T03:51:45.2435112Z 
2022-05-18T03:51:45.2435205Z OK
2022-05-18T03:51:45.2435312Z 
2022-05-18T03:51:45.2435427Z Generating XML reports...
2022-05-18T03:51:45.2469240Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDdpComparisonTest-20220518035143.xml
2022-05-18T03:51:46.0123359Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr0imwjzp
2022-05-18T03:51:46.0124103Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr0imwjzp/_remote_module_non_scriptable.py
2022-05-18T03:51:46.2674231Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T03:51:46.2684454Z 
2022-05-18T03:51:46.2684737Z Running tests...
2022-05-18T03:51:46.2685375Z ----------------------------------------------------------------------
2022-05-18T03:51:46.5867130Z   test_ddp_dist_autograd_sparse_grads (__main__.TensorPipeDdpComparisonTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7294
2022-05-18T03:51:46.5889439Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7295
2022-05-18T03:51:46.5912734Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7296
2022-05-18T03:51:46.5936489Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7297
2022-05-18T03:51:47.1994988Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi3hmf44c
2022-05-18T03:51:47.1995737Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi3hmf44c/_remote_module_non_scriptable.py
2022-05-18T03:51:47.2081593Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkhc2odil
2022-05-18T03:51:47.2082791Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkhc2odil/_remote_module_non_scriptable.py
2022-05-18T03:51:47.2347776Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmdm1_imx
2022-05-18T03:51:47.2348765Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmdm1_imx/_remote_module_non_scriptable.py
2022-05-18T03:51:47.2372286Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgby72zhx
2022-05-18T03:51:47.2374356Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgby72zhx/_remote_module_non_scriptable.py
2022-05-18T03:51:47.4505686Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:51:47.4546336Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:51:47.4832646Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:51:47.4858636Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:51:47.7183294Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:51:47.7184156Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:51:47.7283573Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:51:47.7284443Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:51:47.7285747Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:51:47.7286930Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:51:47.7289656Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:51:47.7290817Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:51:47.7940018Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T03:51:47.7940969Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T03:51:47.7941901Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T03:51:47.7942804Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T03:51:48.1010544Z ok (1.832s)
2022-05-18T03:51:48.1010705Z 
2022-05-18T03:51:48.1011021Z ----------------------------------------------------------------------
2022-05-18T03:51:48.1011259Z Ran 1 test in 1.833s
2022-05-18T03:51:48.1011375Z 
2022-05-18T03:51:48.1011437Z OK
2022-05-18T03:51:48.1011528Z 
2022-05-18T03:51:48.1011624Z Generating XML reports...
2022-05-18T03:51:48.1046396Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDdpComparisonTest-20220518035146.xml
2022-05-18T03:51:48.8723284Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi5zq1itk
2022-05-18T03:51:48.8724284Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi5zq1itk/_remote_module_non_scriptable.py
2022-05-18T03:51:49.1245094Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T03:51:49.1254672Z 
2022-05-18T03:51:49.1254825Z Running tests...
2022-05-18T03:51:49.1255702Z ----------------------------------------------------------------------
2022-05-18T03:51:49.4410225Z   test_backward_ddp_inside (__main__.TensorPipeDdpUnderDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7557
2022-05-18T03:51:49.4433632Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7558
2022-05-18T03:51:49.4456095Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7559
2022-05-18T03:51:49.4480492Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7560
2022-05-18T03:51:49.4504264Z INFO:torch.testing._internal.common_distributed:Started process 4 with pid 7561
2022-05-18T03:51:49.4529439Z INFO:torch.testing._internal.common_distributed:Started process 5 with pid 7562
2022-05-18T03:51:50.2083228Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1r6dgrtx
2022-05-18T03:51:50.2084734Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1r6dgrtx/_remote_module_non_scriptable.py
2022-05-18T03:51:50.2709490Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl2gbrhmh
2022-05-18T03:51:50.2714340Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl2gbrhmh/_remote_module_non_scriptable.py
2022-05-18T03:51:50.2807908Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps8kjokmq
2022-05-18T03:51:50.2810209Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps8kjokmq/_remote_module_non_scriptable.py
2022-05-18T03:51:50.4288569Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj7g6qjj6
2022-05-18T03:51:50.4290303Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj7g6qjj6/_remote_module_non_scriptable.py
2022-05-18T03:51:50.4474648Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7qatg79a
2022-05-18T03:51:50.4478234Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7qatg79a/_remote_module_non_scriptable.py
2022-05-18T03:51:50.4575264Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp41xfglhy
2022-05-18T03:51:50.4576835Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp41xfglhy/_remote_module_non_scriptable.py
2022-05-18T03:51:50.4621799Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 4
2022-05-18T03:51:50.5231350Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 5
2022-05-18T03:51:50.6433721Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:51:50.7242335Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:51:50.7313105Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:51:50.7477080Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:51:51.0300757Z 2022-05-18 03:51:51,029 ddp_under_dist_autograd_test.py:348 INFO p:process 0 t:MainThread: Running the trainer #0...
2022-05-18T03:51:51.0301972Z 2022-05-18 03:51:51,029 ddp_under_dist_autograd_test.py:350 INFO p:process 0 t:MainThread: Initing trainer process group by trainer #0 with ranks [0, 1, 2, 3]
2022-05-18T03:51:51.0397616Z 2022-05-18 03:51:51,039 ddp_under_dist_autograd_test.py:348 INFO p:process 1 t:MainThread: Running the trainer #1...
2022-05-18T03:51:51.0398805Z 2022-05-18 03:51:51,039 ddp_under_dist_autograd_test.py:350 INFO p:process 1 t:MainThread: Initing trainer process group by trainer #1 with ranks [0, 1, 2, 3]
2022-05-18T03:51:51.0407707Z 2022-05-18 03:51:51,040 ddp_under_dist_autograd_test.py:348 INFO p:process 2 t:MainThread: Running the trainer #2...
2022-05-18T03:51:51.0408895Z 2022-05-18 03:51:51,040 ddp_under_dist_autograd_test.py:350 INFO p:process 2 t:MainThread: Initing trainer process group by trainer #2 with ranks [0, 1, 2, 3]
2022-05-18T03:51:51.0596171Z 2022-05-18 03:51:51,058 ddp_under_dist_autograd_test.py:329 INFO p:process 4 t:MainThread: The remote worker is running.
2022-05-18T03:51:51.0602655Z 2022-05-18 03:51:51,059 ddp_under_dist_autograd_test.py:348 INFO p:process 3 t:MainThread: Running the trainer #3...
2022-05-18T03:51:51.0611601Z 2022-05-18 03:51:51,060 ddp_under_dist_autograd_test.py:368 INFO p:process 5 t:MainThread: Running the master process...
2022-05-18T03:51:51.0612786Z 2022-05-18 03:51:51,060 ddp_under_dist_autograd_test.py:350 INFO p:process 3 t:MainThread: Initing trainer process group by trainer #3 with ranks [0, 1, 2, 3]
2022-05-18T03:51:51.0930032Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:51:51.1034832Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:51:51.1136206Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:51:51.1258107Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:51:51.1259027Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 4
2022-05-18T03:51:51.1259848Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 5
2022-05-18T03:51:51.1261006Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:1 with 6 nodes.
2022-05-18T03:51:51.1262170Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 6 nodes.
2022-05-18T03:51:51.1263450Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 6 nodes.
2022-05-18T03:51:51.1264597Z 2022-05-18 03:51:51,124 ddp_under_dist_autograd_test.py:359 INFO p:process 0 t:MainThread: Waiting for shutdown signal on trainer #0...
2022-05-18T03:51:51.1265698Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 6 nodes.
2022-05-18T03:51:51.1266821Z 2022-05-18 03:51:51,125 ddp_under_dist_autograd_test.py:359 INFO p:process 3 t:MainThread: Waiting for shutdown signal on trainer #3...
2022-05-18T03:51:51.1267914Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 6 nodes.
2022-05-18T03:51:51.1269023Z 2022-05-18 03:51:51,125 ddp_under_dist_autograd_test.py:359 INFO p:process 2 t:MainThread: Waiting for shutdown signal on trainer #2...
2022-05-18T03:51:51.1344506Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:1 with 6 nodes.
2022-05-18T03:51:51.1345375Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 4
2022-05-18T03:51:51.1360145Z 2022-05-18 03:51:51,134 ddp_under_dist_autograd_test.py:359 INFO p:process 1 t:MainThread: Waiting for shutdown signal on trainer #1...
2022-05-18T03:51:51.1361217Z 2022-05-18 03:51:51,135 ddp_under_dist_autograd_test.py:382 INFO p:process 5 t:MainThread: Created remote rrefs on master
2022-05-18T03:51:51.1366067Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 5
2022-05-18T03:51:51.1716007Z 2022-05-18 03:51:51,170 ddp_under_dist_autograd_test.py:94 INFO p:process 4 t:Dummy-2: Initing RemoteEM with 2 3
2022-05-18T03:51:51.1720886Z 2022-05-18 03:51:51,171 ddp_under_dist_autograd_test.py:120 INFO p:process 4 t:Dummy-3: Initing RemoteNet with 5 3
2022-05-18T03:51:51.3018200Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T03:51:51.3118763Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T03:51:51.3226744Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-05-18T03:51:51.3227982Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 3
2022-05-18T03:51:51.3229597Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:2 with 6 nodes.
2022-05-18T03:51:51.3231105Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 6 nodes.
2022-05-18T03:51:51.3236724Z 2022-05-18 03:51:51,323 ddp_under_dist_autograd_test.py:153 INFO p:process 0 t:Dummy-2: Use DDP for the second local net.
2022-05-18T03:51:51.3248149Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 6 nodes.
2022-05-18T03:51:51.3268895Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:2 with 6 nodes.
2022-05-18T03:51:51.3270367Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:2 with 6 nodes.
2022-05-18T03:51:51.3289883Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 6 nodes.
2022-05-18T03:51:51.3299533Z 2022-05-18 03:51:51,329 ddp_under_dist_autograd_test.py:153 INFO p:process 2 t:Dummy-2: Use DDP for the second local net.
2022-05-18T03:51:51.3309850Z 2022-05-18 03:51:51,330 ddp_under_dist_autograd_test.py:153 INFO p:process 1 t:Dummy-2: Use DDP for the second local net.
2022-05-18T03:51:51.3375036Z 2022-05-18 03:51:51,337 ddp_under_dist_autograd_test.py:153 INFO p:process 3 t:Dummy-2: Use DDP for the second local net.
2022-05-18T03:51:51.3445860Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:621: UserWarning: The `check_reduction` argument in `DistributedDataParallel` module is deprecated. Please avoid using it.
2022-05-18T03:51:51.3446527Z   "The `check_reduction` argument in `DistributedDataParallel` "
2022-05-18T03:51:51.3531459Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:621: UserWarning: The `check_reduction` argument in `DistributedDataParallel` module is deprecated. Please avoid using it.
2022-05-18T03:51:51.3532428Z   "The `check_reduction` argument in `DistributedDataParallel` "
2022-05-18T03:51:51.3579464Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:621: UserWarning: The `check_reduction` argument in `DistributedDataParallel` module is deprecated. Please avoid using it.
2022-05-18T03:51:51.3580417Z   "The `check_reduction` argument in `DistributedDataParallel` "
2022-05-18T03:51:51.3615442Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:621: UserWarning: The `check_reduction` argument in `DistributedDataParallel` module is deprecated. Please avoid using it.
2022-05-18T03:51:51.3616385Z   "The `check_reduction` argument in `DistributedDataParallel` "
2022-05-18T03:51:51.3638222Z 2022-05-18 03:51:51,363 ddp_under_dist_autograd_test.py:159 INFO p:process 1 t:Dummy-2: HybridModel has 2 groups of parameters.
2022-05-18T03:51:51.3640716Z 2022-05-18 03:51:51,363 ddp_under_dist_autograd_test.py:159 INFO p:process 0 t:Dummy-2: HybridModel has 2 groups of parameters.
2022-05-18T03:51:51.3641961Z 2022-05-18 03:51:51,363 ddp_under_dist_autograd_test.py:211 INFO p:process 0 t:Dummy-2: Succeeded in creating a HybridModel instance with 1 ddp params and 1 other local params.
2022-05-18T03:51:51.3643107Z 2022-05-18 03:51:51,363 ddp_under_dist_autograd_test.py:159 INFO p:process 3 t:Dummy-2: HybridModel has 2 groups of parameters.
2022-05-18T03:51:51.3646468Z 2022-05-18 03:51:51,363 ddp_under_dist_autograd_test.py:211 INFO p:process 1 t:Dummy-2: Succeeded in creating a HybridModel instance with 1 ddp params and 1 other local params.
2022-05-18T03:51:51.3647799Z 2022-05-18 03:51:51,363 ddp_under_dist_autograd_test.py:211 INFO p:process 3 t:Dummy-2: Succeeded in creating a HybridModel instance with 1 ddp params and 1 other local params.
2022-05-18T03:51:51.3648931Z 2022-05-18 03:51:51,364 ddp_under_dist_autograd_test.py:159 INFO p:process 2 t:Dummy-2: HybridModel has 2 groups of parameters.
2022-05-18T03:51:51.3650141Z 2022-05-18 03:51:51,364 ddp_under_dist_autograd_test.py:211 INFO p:process 2 t:Dummy-2: Succeeded in creating a HybridModel instance with 1 ddp params and 1 other local params.
2022-05-18T03:51:51.4247476Z 2022-05-18 03:51:51,424 ddp_under_dist_autograd_test.py:261 INFO p:process 1 t:Dummy-3: Loss is -56.0 for mini batch: FeatureSet(dense_features=tensor([[-1., -1.],
2022-05-18T03:51:51.4248253Z         [-1., -1.],
2022-05-18T03:51:51.4248681Z         [-1.,  1.],
2022-05-18T03:51:51.4250280Z         [-1.,  1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([-1., -1., -1., -1.])). Grads dict has 2 entries: {Parameter containing:
2022-05-18T03:51:51.4251310Z tensor([[-1.,  1.,  1.]], requires_grad=True): tensor([[0., 0., 0.]]), Parameter containing:
2022-05-18T03:51:51.4252025Z tensor([[-1.,  1.],
2022-05-18T03:51:51.4252441Z         [ 1.,  1.]], requires_grad=True): tensor([[ 16., -16.],
2022-05-18T03:51:51.4252815Z         [ 16., -16.]])}
2022-05-18T03:51:51.4253899Z 2022-05-18 03:51:51,424 ddp_under_dist_autograd_test.py:261 INFO p:process 3 t:Dummy-3: Loss is 0.0 for mini batch: FeatureSet(dense_features=tensor([[ 1.,  1.],
2022-05-18T03:51:51.4254333Z         [ 1.,  1.],
2022-05-18T03:51:51.4254641Z         [ 1., -1.],
2022-05-18T03:51:51.4255171Z         [ 1., -1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([1., 1., 1., 1.])). Grads dict has 2 entries: {Parameter containing:
2022-05-18T03:51:51.4255794Z tensor([[-1.,  1.,  1.]], requires_grad=True): tensor([[0., 0., 0.]]), Parameter containing:
2022-05-18T03:51:51.4256255Z tensor([[-1.,  1.],
2022-05-18T03:51:51.4256710Z         [ 1.,  1.]], requires_grad=True): tensor([[32., 16.],
2022-05-18T03:51:51.4257134Z         [ 0., 16.]])}
2022-05-18T03:51:51.4258472Z 2022-05-18 03:51:51,424 ddp_under_dist_autograd_test.py:261 INFO p:process 0 t:Dummy-3: Loss is 0.0 for mini batch: FeatureSet(dense_features=tensor([[ 1., -1.],
2022-05-18T03:51:51.4259278Z         [ 1., -1.],
2022-05-18T03:51:51.4259637Z         [ 1.,  1.],
2022-05-18T03:51:51.4260509Z         [ 1.,  1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([-1., -1., -1., -1.])). Grads dict has 2 entries: {Parameter containing:
2022-05-18T03:51:51.4261396Z tensor([[-1.,  1.,  1.]], requires_grad=True): tensor([[0., 0., 0.]]), Parameter containing:
2022-05-18T03:51:51.4261956Z tensor([[-1.,  1.],
2022-05-18T03:51:51.4262539Z         [ 1.,  1.]], requires_grad=True): tensor([[-32., -16.],
2022-05-18T03:51:51.4263185Z         [  0., -16.]])}
2022-05-18T03:51:51.4274219Z 2022-05-18 03:51:51,426 ddp_under_dist_autograd_test.py:261 INFO p:process 2 t:Dummy-3: Loss is 56.0 for mini batch: FeatureSet(dense_features=tensor([[-1.,  1.],
2022-05-18T03:51:51.4274925Z         [-1.,  1.],
2022-05-18T03:51:51.4275350Z         [-1., -1.],
2022-05-18T03:51:51.4276191Z         [-1., -1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([1., 1., 1., 1.])). Grads dict has 2 entries: {Parameter containing:
2022-05-18T03:51:51.4277105Z tensor([[-1.,  1.,  1.]], requires_grad=True): tensor([[0., 0., 0.]]), Parameter containing:
2022-05-18T03:51:51.4277663Z tensor([[-1.,  1.],
2022-05-18T03:51:51.4278221Z         [ 1.,  1.]], requires_grad=True): tensor([[-16.,  16.],
2022-05-18T03:51:51.4278723Z         [-16.,  16.]])}
2022-05-18T03:51:51.4501413Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T03:51:51.4512798Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T03:51:51.4523732Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T03:51:51.4527228Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T03:51:51.4570756Z 2022-05-18 03:51:51,456 ddp_under_dist_autograd_test.py:261 INFO p:process 3 t:Dummy-4: Loss is 0.0 for mini batch: FeatureSet(dense_features=tensor([[ 1.,  1.],
2022-05-18T03:51:51.4571594Z         [ 1.,  1.],
2022-05-18T03:51:51.4572033Z         [ 1., -1.],
2022-05-18T03:51:51.4572897Z         [ 1., -1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([1., 1., 1., 1.])). Grads dict has 2 entries: {Parameter containing:
2022-05-18T03:51:51.4573791Z tensor([[-1.,  1.,  1.]], requires_grad=True): tensor([[0., 0., 0.]]), Parameter containing:
2022-05-18T03:51:51.4574358Z tensor([[-1.,  1.],
2022-05-18T03:51:51.4574797Z         [ 1.,  1.]], requires_grad=True): tensor([[32., 16.],
2022-05-18T03:51:51.4575223Z         [ 0., 16.]])}
2022-05-18T03:51:51.4595313Z 2022-05-18 03:51:51,459 ddp_under_dist_autograd_test.py:261 INFO p:process 0 t:Dummy-4: Loss is 0.0 for mini batch: FeatureSet(dense_features=tensor([[ 1., -1.],
2022-05-18T03:51:51.4595983Z         [ 1., -1.],
2022-05-18T03:51:51.4596348Z         [ 1.,  1.],
2022-05-18T03:51:51.4597524Z         [ 1.,  1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([-1., -1., -1., -1.])). Grads dict has 2 entries: {Parameter containing:
2022-05-18T03:51:51.4598515Z tensor([[-1.,  1.,  1.]], requires_grad=True): tensor([[0., 0., 0.]]), Parameter containing:
2022-05-18T03:51:51.4599063Z tensor([[-1.,  1.],
2022-05-18T03:51:51.4599643Z         [ 1.,  1.]], requires_grad=True): tensor([[-32., -16.],
2022-05-18T03:51:51.4600136Z         [  0., -16.]])}
2022-05-18T03:51:51.4618252Z 2022-05-18 03:51:51,461 ddp_under_dist_autograd_test.py:261 INFO p:process 2 t:Dummy-4: Loss is 56.0 for mini batch: FeatureSet(dense_features=tensor([[-1.,  1.],
2022-05-18T03:51:51.4618744Z         [-1.,  1.],
2022-05-18T03:51:51.4619053Z         [-1., -1.],
2022-05-18T03:51:51.4619701Z         [-1., -1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([1., 1., 1., 1.])). Grads dict has 2 entries: {Parameter containing:
2022-05-18T03:51:51.4620577Z tensor([[-1.,  1.,  1.]], requires_grad=True): tensor([[0., 0., 0.]]), Parameter containing:
2022-05-18T03:51:51.4621143Z tensor([[-1.,  1.],
2022-05-18T03:51:51.4621724Z         [ 1.,  1.]], requires_grad=True): tensor([[-16.,  16.],
2022-05-18T03:51:51.4622215Z         [-16.,  16.]])}
2022-05-18T03:51:51.4641942Z 2022-05-18 03:51:51,463 ddp_under_dist_autograd_test.py:261 INFO p:process 1 t:Dummy-4: Loss is -56.0 for mini batch: FeatureSet(dense_features=tensor([[-1., -1.],
2022-05-18T03:51:51.4642678Z         [-1., -1.],
2022-05-18T03:51:51.4643084Z         [-1.,  1.],
2022-05-18T03:51:51.4643901Z         [-1.,  1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([-1., -1., -1., -1.])). Grads dict has 2 entries: {Parameter containing:
2022-05-18T03:51:51.4644820Z tensor([[-1.,  1.,  1.]], requires_grad=True): tensor([[0., 0., 0.]]), Parameter containing:
2022-05-18T03:51:51.4645396Z tensor([[-1.,  1.],
2022-05-18T03:51:51.4645982Z         [ 1.,  1.]], requires_grad=True): tensor([[ 16., -16.],
2022-05-18T03:51:51.4646466Z         [ 16., -16.]])}
2022-05-18T03:51:51.4807345Z 2022-05-18 03:51:51,480 ddp_under_dist_autograd_test.py:261 INFO p:process 1 t:Dummy-5: Loss is -56.0 for mini batch: FeatureSet(dense_features=tensor([[-1., -1.],
2022-05-18T03:51:51.4807814Z         [-1., -1.],
2022-05-18T03:51:51.4808003Z         [-1.,  1.],
2022-05-18T03:51:51.4808442Z         [-1.,  1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([-1., -1., -1., -1.])). Grads dict has 2 entries: {Parameter containing:
2022-05-18T03:51:51.4808873Z tensor([[-1.,  1.,  1.]], requires_grad=True): tensor([[0., 0., 0.]]), Parameter containing:
2022-05-18T03:51:51.4809170Z tensor([[-1.,  1.],
2022-05-18T03:51:51.4809556Z         [ 1.,  1.]], requires_grad=True): tensor([[ 16., -16.],
2022-05-18T03:51:51.4809894Z         [ 16., -16.]])}
2022-05-18T03:51:51.4810382Z 2022-05-18 03:51:51,480 ddp_under_dist_autograd_test.py:261 INFO p:process 2 t:Dummy-5: Loss is 56.0 for mini batch: FeatureSet(dense_features=tensor([[-1.,  1.],
2022-05-18T03:51:51.4810702Z         [-1.,  1.],
2022-05-18T03:51:51.4810942Z         [-1., -1.],
2022-05-18T03:51:51.4811325Z         [-1., -1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([1., 1., 1., 1.])). Grads dict has 2 entries: {Parameter containing:
2022-05-18T03:51:51.4811859Z tensor([[-1.,  1.,  1.]], requires_grad=True): tensor([[0., 0., 0.]]), Parameter containing:
2022-05-18T03:51:51.4812263Z tensor([[-1.,  1.],
2022-05-18T03:51:51.4812592Z         [ 1.,  1.]], requires_grad=True): tensor([[-16.,  16.],
2022-05-18T03:51:51.4812831Z         [-16.,  16.]])}
2022-05-18T03:51:51.4813468Z 2022-05-18 03:51:51,480 ddp_under_dist_autograd_test.py:261 INFO p:process 0 t:Dummy-5: Loss is 0.0 for mini batch: FeatureSet(dense_features=tensor([[ 1., -1.],
2022-05-18T03:51:51.4813816Z         [ 1., -1.],
2022-05-18T03:51:51.4814053Z         [ 1.,  1.],
2022-05-18T03:51:51.4814643Z         [ 1.,  1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([-1., -1., -1., -1.])). Grads dict has 2 entries: {Parameter containing:
2022-05-18T03:51:51.4815164Z tensor([[-1.,  1.,  1.]], requires_grad=True): tensor([[0., 0., 0.]]), Parameter containing:
2022-05-18T03:51:51.4815684Z tensor([[-1.,  1.],
2022-05-18T03:51:51.4816013Z         [ 1.,  1.]], requires_grad=True): tensor([[-32., -16.],
2022-05-18T03:51:51.4816288Z         [  0., -16.]])}
2022-05-18T03:51:51.4838330Z 2022-05-18 03:51:51,483 ddp_under_dist_autograd_test.py:261 INFO p:process 3 t:Dummy-5: Loss is 0.0 for mini batch: FeatureSet(dense_features=tensor([[ 1.,  1.],
2022-05-18T03:51:51.4838844Z         [ 1.,  1.],
2022-05-18T03:51:51.4839197Z         [ 1., -1.],
2022-05-18T03:51:51.4839906Z         [ 1., -1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([1., 1., 1., 1.])). Grads dict has 2 entries: {Parameter containing:
2022-05-18T03:51:51.4840559Z tensor([[-1.,  1.,  1.]], requires_grad=True): tensor([[0., 0., 0.]]), Parameter containing:
2022-05-18T03:51:51.4840866Z tensor([[-1.,  1.],
2022-05-18T03:51:51.4841168Z         [ 1.,  1.]], requires_grad=True): tensor([[32., 16.],
2022-05-18T03:51:51.4841481Z         [ 0., 16.]])}
2022-05-18T03:51:51.4879445Z 2022-05-18 03:51:51,487 ddp_under_dist_autograd_test.py:364 INFO p:process 0 t:MainThread: Exiting the trainer #0...
2022-05-18T03:51:51.4891822Z 2022-05-18 03:51:51,488 ddp_under_dist_autograd_test.py:364 INFO p:process 1 t:MainThread: Exiting the trainer #1...
2022-05-18T03:51:51.4893556Z 2022-05-18 03:51:51,489 ddp_under_dist_autograd_test.py:364 INFO p:process 2 t:MainThread: Exiting the trainer #2...
2022-05-18T03:51:51.4899888Z 2022-05-18 03:51:51,489 ddp_under_dist_autograd_test.py:364 INFO p:process 3 t:MainThread: Exiting the trainer #3...
2022-05-18T03:51:51.4908569Z 2022-05-18 03:51:51,490 ddp_under_dist_autograd_test.py:344 INFO p:process 4 t:MainThread: Exiting remote worker.
2022-05-18T03:51:51.8594206Z ok (2.734s)
2022-05-18T03:51:51.8594439Z 
2022-05-18T03:51:51.8594746Z ----------------------------------------------------------------------
2022-05-18T03:51:51.8595000Z Ran 1 test in 2.734s
2022-05-18T03:51:51.8595102Z 
2022-05-18T03:51:51.8595182Z OK
2022-05-18T03:51:51.8595275Z 
2022-05-18T03:51:51.8595371Z Generating XML reports...
2022-05-18T03:51:51.8629142Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDdpUnderDistAutogradTest-20220518035149.xml
2022-05-18T03:51:52.6319771Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgk6pocnp
2022-05-18T03:51:52.6321326Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgk6pocnp/_remote_module_non_scriptable.py
2022-05-18T03:51:52.8853478Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T03:51:52.8863466Z 
2022-05-18T03:51:52.8863572Z Running tests...
2022-05-18T03:51:52.8864027Z ----------------------------------------------------------------------
2022-05-18T03:51:53.2008016Z   test_backward_ddp_outside (__main__.TensorPipeDdpUnderDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7965
2022-05-18T03:51:53.2031820Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7966
2022-05-18T03:51:53.2054781Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7967
2022-05-18T03:51:53.2079779Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7968
2022-05-18T03:51:53.2104209Z INFO:torch.testing._internal.common_distributed:Started process 4 with pid 7969
2022-05-18T03:51:53.2129602Z INFO:torch.testing._internal.common_distributed:Started process 5 with pid 7970
2022-05-18T03:51:54.0074511Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpquxs1aoc
2022-05-18T03:51:54.0075433Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpquxs1aoc/_remote_module_non_scriptable.py
2022-05-18T03:51:54.0334425Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1lj9ace1
2022-05-18T03:51:54.0335862Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1lj9ace1/_remote_module_non_scriptable.py
2022-05-18T03:51:54.0789560Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0s28dh7a
2022-05-18T03:51:54.0790961Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0s28dh7a/_remote_module_non_scriptable.py
2022-05-18T03:51:54.0818430Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp56_sc4ih
2022-05-18T03:51:54.0821968Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp56_sc4ih/_remote_module_non_scriptable.py
2022-05-18T03:51:54.0988871Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5xubagtd
2022-05-18T03:51:54.0991833Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5xubagtd/_remote_module_non_scriptable.py
2022-05-18T03:51:54.1228430Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpevxg84s_
2022-05-18T03:51:54.1230901Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpevxg84s_/_remote_module_non_scriptable.py
2022-05-18T03:51:54.3103210Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:51:54.3268654Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:51:54.4176292Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 5
2022-05-18T03:51:54.4352463Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:51:54.4731315Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 4
2022-05-18T03:51:54.4750009Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:51:54.7413812Z 2022-05-18 03:51:54,740 ddp_under_dist_autograd_test.py:348 INFO p:process 0 t:MainThread: Running the trainer #0...
2022-05-18T03:51:54.7419649Z 2022-05-18 03:51:54,740 ddp_under_dist_autograd_test.py:350 INFO p:process 0 t:MainThread: Initing trainer process group by trainer #0 with ranks [0, 1, 2, 3]
2022-05-18T03:51:54.7513753Z 2022-05-18 03:51:54,750 ddp_under_dist_autograd_test.py:348 INFO p:process 1 t:MainThread: Running the trainer #1...
2022-05-18T03:51:54.7514956Z 2022-05-18 03:51:54,750 ddp_under_dist_autograd_test.py:350 INFO p:process 1 t:MainThread: Initing trainer process group by trainer #1 with ranks [0, 1, 2, 3]
2022-05-18T03:51:54.7713860Z 2022-05-18 03:51:54,770 ddp_under_dist_autograd_test.py:348 INFO p:process 2 t:MainThread: Running the trainer #2...
2022-05-18T03:51:54.7719682Z 2022-05-18 03:51:54,771 ddp_under_dist_autograd_test.py:350 INFO p:process 2 t:MainThread: Initing trainer process group by trainer #2 with ranks [0, 1, 2, 3]
2022-05-18T03:51:54.7811500Z 2022-05-18 03:51:54,780 ddp_under_dist_autograd_test.py:348 INFO p:process 3 t:MainThread: Running the trainer #3...
2022-05-18T03:51:54.7815623Z 2022-05-18 03:51:54,780 ddp_under_dist_autograd_test.py:350 INFO p:process 3 t:MainThread: Initing trainer process group by trainer #3 with ranks [0, 1, 2, 3]
2022-05-18T03:51:54.7824270Z 2022-05-18 03:51:54,781 ddp_under_dist_autograd_test.py:329 INFO p:process 4 t:MainThread: The remote worker is running.
2022-05-18T03:51:54.7830302Z 2022-05-18 03:51:54,780 ddp_under_dist_autograd_test.py:368 INFO p:process 5 t:MainThread: Running the master process...
2022-05-18T03:51:54.7951713Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:51:54.8053462Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:51:54.8058318Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:51:54.8167311Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:51:54.8268518Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 5
2022-05-18T03:51:54.8272957Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 4
2022-05-18T03:51:54.8279478Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:1 with 6 nodes.
2022-05-18T03:51:54.8367224Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 6 nodes.
2022-05-18T03:51:54.8373139Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 6 nodes.
2022-05-18T03:51:54.8378055Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:1 with 6 nodes.
2022-05-18T03:51:54.8384646Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 6 nodes.
2022-05-18T03:51:54.8391018Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 6 nodes.
2022-05-18T03:51:54.8396704Z 2022-05-18 03:51:54,838 ddp_under_dist_autograd_test.py:359 INFO p:process 0 t:MainThread: Waiting for shutdown signal on trainer #0...
2022-05-18T03:51:54.8402101Z 2022-05-18 03:51:54,839 ddp_under_dist_autograd_test.py:359 INFO p:process 1 t:MainThread: Waiting for shutdown signal on trainer #1...
2022-05-18T03:51:54.8409074Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 4
2022-05-18T03:51:54.8469570Z 2022-05-18 03:51:54,846 ddp_under_dist_autograd_test.py:359 INFO p:process 2 t:MainThread: Waiting for shutdown signal on trainer #2...
2022-05-18T03:51:54.8479086Z 2022-05-18 03:51:54,847 ddp_under_dist_autograd_test.py:359 INFO p:process 3 t:MainThread: Waiting for shutdown signal on trainer #3...
2022-05-18T03:51:54.8489910Z 2022-05-18 03:51:54,848 ddp_under_dist_autograd_test.py:382 INFO p:process 5 t:MainThread: Created remote rrefs on master
2022-05-18T03:51:54.8501891Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 5
2022-05-18T03:51:54.8809820Z 2022-05-18 03:51:54,880 ddp_under_dist_autograd_test.py:94 INFO p:process 4 t:Dummy-2: Initing RemoteEM with 2 3
2022-05-18T03:51:54.8819402Z 2022-05-18 03:51:54,880 ddp_under_dist_autograd_test.py:120 INFO p:process 4 t:Dummy-3: Initing RemoteNet with 5 3
2022-05-18T03:51:55.0233459Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T03:51:55.0239918Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T03:51:55.0333914Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 3
2022-05-18T03:51:55.0337580Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-05-18T03:51:55.0338901Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 6 nodes.
2022-05-18T03:51:55.0340573Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:2 with 6 nodes.
2022-05-18T03:51:55.0341784Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:2 with 6 nodes.
2022-05-18T03:51:55.0350854Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 6 nodes.
2022-05-18T03:51:55.0352729Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 6 nodes.
2022-05-18T03:51:55.0366933Z 2022-05-18 03:51:55,036 ddp_under_dist_autograd_test.py:159 INFO p:process 0 t:Dummy-2: HybridModel has 2 groups of parameters.
2022-05-18T03:51:55.0368001Z 2022-05-18 03:51:55,036 ddp_under_dist_autograd_test.py:202 INFO p:process 0 t:Dummy-2: Wrapping the whole hybrid module into DDP.
2022-05-18T03:51:55.0424615Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:2 with 6 nodes.
2022-05-18T03:51:55.0453576Z 2022-05-18 03:51:55,044 ddp_under_dist_autograd_test.py:159 INFO p:process 2 t:Dummy-2: HybridModel has 2 groups of parameters.
2022-05-18T03:51:55.0454838Z 2022-05-18 03:51:55,045 ddp_under_dist_autograd_test.py:159 INFO p:process 3 t:Dummy-2: HybridModel has 2 groups of parameters.
2022-05-18T03:51:55.0455914Z 2022-05-18 03:51:55,045 ddp_under_dist_autograd_test.py:202 INFO p:process 3 t:Dummy-2: Wrapping the whole hybrid module into DDP.
2022-05-18T03:51:55.0457505Z 2022-05-18 03:51:55,045 ddp_under_dist_autograd_test.py:202 INFO p:process 2 t:Dummy-2: Wrapping the whole hybrid module into DDP.
2022-05-18T03:51:55.0470943Z 2022-05-18 03:51:55,046 ddp_under_dist_autograd_test.py:159 INFO p:process 1 t:Dummy-2: HybridModel has 2 groups of parameters.
2022-05-18T03:51:55.0472855Z 2022-05-18 03:51:55,046 ddp_under_dist_autograd_test.py:202 INFO p:process 1 t:Dummy-2: Wrapping the whole hybrid module into DDP.
2022-05-18T03:51:55.0668862Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:621: UserWarning: The `check_reduction` argument in `DistributedDataParallel` module is deprecated. Please avoid using it.
2022-05-18T03:51:55.0669888Z   "The `check_reduction` argument in `DistributedDataParallel` "
2022-05-18T03:51:55.0690639Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:621: UserWarning: The `check_reduction` argument in `DistributedDataParallel` module is deprecated. Please avoid using it.
2022-05-18T03:51:55.0691599Z   "The `check_reduction` argument in `DistributedDataParallel` "
2022-05-18T03:51:55.0730165Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:621: UserWarning: The `check_reduction` argument in `DistributedDataParallel` module is deprecated. Please avoid using it.
2022-05-18T03:51:55.0731139Z   "The `check_reduction` argument in `DistributedDataParallel` "
2022-05-18T03:51:55.0744856Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:621: UserWarning: The `check_reduction` argument in `DistributedDataParallel` module is deprecated. Please avoid using it.
2022-05-18T03:51:55.0745835Z   "The `check_reduction` argument in `DistributedDataParallel` "
2022-05-18T03:51:55.0769975Z 2022-05-18 03:51:55,076 ddp_under_dist_autograd_test.py:211 INFO p:process 1 t:Dummy-2: Succeeded in creating a HybridModel instance with 2 ddp params and 0 other local params.
2022-05-18T03:51:55.0770934Z 2022-05-18 03:51:55,076 ddp_under_dist_autograd_test.py:211 INFO p:process 3 t:Dummy-2: Succeeded in creating a HybridModel instance with 2 ddp params and 0 other local params.
2022-05-18T03:51:55.0780072Z 2022-05-18 03:51:55,077 ddp_under_dist_autograd_test.py:211 INFO p:process 0 t:Dummy-2: Succeeded in creating a HybridModel instance with 2 ddp params and 0 other local params.
2022-05-18T03:51:55.0798964Z 2022-05-18 03:51:55,079 ddp_under_dist_autograd_test.py:211 INFO p:process 2 t:Dummy-2: Succeeded in creating a HybridModel instance with 2 ddp params and 0 other local params.
2022-05-18T03:51:55.1194902Z 2022-05-18 03:51:55,118 ddp_under_dist_autograd_test.py:261 INFO p:process 2 t:Dummy-3: Loss is 56.0 for mini batch: FeatureSet(dense_features=tensor([[-1.,  1.],
2022-05-18T03:51:55.1195465Z         [-1.,  1.],
2022-05-18T03:51:55.1202197Z         [-1., -1.],
2022-05-18T03:51:55.1203118Z         [-1., -1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([1., 1., 1., 1.])). Grads dict has 2 entries: {Parameter containing:
2022-05-18T03:51:55.1204013Z tensor([[-1.,  1.,  1.]], requires_grad=True): tensor([[0., 0., 0.]]), Parameter containing:
2022-05-18T03:51:55.1204497Z tensor([[-1.,  1.],
2022-05-18T03:51:55.1204901Z         [ 1.,  1.]], requires_grad=True): tensor([[0., 0.],
2022-05-18T03:51:55.1205329Z         [0., 0.]])}
2022-05-18T03:51:55.1206249Z 2022-05-18 03:51:55,119 ddp_under_dist_autograd_test.py:261 INFO p:process 0 t:Dummy-3: Loss is 0.0 for mini batch: FeatureSet(dense_features=tensor([[ 1., -1.],
2022-05-18T03:51:55.1206972Z         [ 1., -1.],
2022-05-18T03:51:55.1207323Z         [ 1.,  1.],
2022-05-18T03:51:55.1208531Z         [ 1.,  1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([-1., -1., -1., -1.])). Grads dict has 2 entries: {Parameter containing:
2022-05-18T03:51:55.1209559Z tensor([[-1.,  1.,  1.]], requires_grad=True): tensor([[0., 0., 0.]]), Parameter containing:
2022-05-18T03:51:55.1210138Z tensor([[-1.,  1.],
2022-05-18T03:51:55.1210589Z         [ 1.,  1.]], requires_grad=True): tensor([[0., 0.],
2022-05-18T03:51:55.1210992Z         [0., 0.]])}
2022-05-18T03:51:55.1220784Z 2022-05-18 03:51:55,121 ddp_under_dist_autograd_test.py:261 INFO p:process 3 t:Dummy-3: Loss is 0.0 for mini batch: FeatureSet(dense_features=tensor([[ 1.,  1.],
2022-05-18T03:51:55.1221447Z         [ 1.,  1.],
2022-05-18T03:51:55.1221865Z         [ 1., -1.],
2022-05-18T03:51:55.1222727Z         [ 1., -1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([1., 1., 1., 1.])). Grads dict has 2 entries: {Parameter containing:
2022-05-18T03:51:55.1223741Z tensor([[-1.,  1.,  1.]], requires_grad=True): tensor([[0., 0., 0.]]), Parameter containing:
2022-05-18T03:51:55.1224259Z tensor([[-1.,  1.],
2022-05-18T03:51:55.1224695Z         [ 1.,  1.]], requires_grad=True): tensor([[0., 0.],
2022-05-18T03:51:55.1225143Z         [0., 0.]])}
2022-05-18T03:51:55.1253343Z 2022-05-18 03:51:55,124 ddp_under_dist_autograd_test.py:261 INFO p:process 1 t:Dummy-3: Loss is -56.0 for mini batch: FeatureSet(dense_features=tensor([[-1., -1.],
2022-05-18T03:51:55.1253996Z         [-1., -1.],
2022-05-18T03:51:55.1254351Z         [-1.,  1.],
2022-05-18T03:51:55.1255094Z         [-1.,  1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([-1., -1., -1., -1.])). Grads dict has 2 entries: {Parameter containing:
2022-05-18T03:51:55.1255908Z tensor([[-1.,  1.,  1.]], requires_grad=True): tensor([[0., 0., 0.]]), Parameter containing:
2022-05-18T03:51:55.1256475Z tensor([[-1.,  1.],
2022-05-18T03:51:55.1256938Z         [ 1.,  1.]], requires_grad=True): tensor([[0., 0.],
2022-05-18T03:51:55.1257366Z         [0., 0.]])}
2022-05-18T03:51:55.1304063Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T03:51:55.1320981Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T03:51:55.1321680Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T03:51:55.1336467Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T03:51:55.1467388Z 2022-05-18 03:51:55,146 ddp_under_dist_autograd_test.py:261 INFO p:process 1 t:Dummy-4: Loss is -56.0 for mini batch: FeatureSet(dense_features=tensor([[-1., -1.],
2022-05-18T03:51:55.1468042Z         [-1., -1.],
2022-05-18T03:51:55.1468388Z         [-1.,  1.],
2022-05-18T03:51:55.1469031Z         [-1.,  1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([-1., -1., -1., -1.])). Grads dict has 2 entries: {Parameter containing:
2022-05-18T03:51:55.1469791Z tensor([[-1.,  1.,  1.]], requires_grad=True): tensor([[0., 0., 0.]]), Parameter containing:
2022-05-18T03:51:55.1470290Z tensor([[-1.,  1.],
2022-05-18T03:51:55.1470663Z         [ 1.,  1.]], requires_grad=True): tensor([[0., 0.],
2022-05-18T03:51:55.1471062Z         [0., 0.]])}
2022-05-18T03:51:55.1472008Z 2022-05-18 03:51:55,146 ddp_under_dist_autograd_test.py:261 INFO p:process 0 t:Dummy-4: Loss is 0.0 for mini batch: FeatureSet(dense_features=tensor([[ 1., -1.],
2022-05-18T03:51:55.1472693Z         [ 1., -1.],
2022-05-18T03:51:55.1473008Z         [ 1.,  1.],
2022-05-18T03:51:55.1473856Z         [ 1.,  1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([-1., -1., -1., -1.])). Grads dict has 2 entries: {Parameter containing:
2022-05-18T03:51:55.1474735Z tensor([[-1.,  1.,  1.]], requires_grad=True): tensor([[0., 0., 0.]]), Parameter containing:
2022-05-18T03:51:55.1475297Z tensor([[-1.,  1.],
2022-05-18T03:51:55.1475718Z         [ 1.,  1.]], requires_grad=True): tensor([[0., 0.],
2022-05-18T03:51:55.1476132Z         [0., 0.]])}
2022-05-18T03:51:55.1477062Z 2022-05-18 03:51:55,146 ddp_under_dist_autograd_test.py:261 INFO p:process 2 t:Dummy-4: Loss is 56.0 for mini batch: FeatureSet(dense_features=tensor([[-1.,  1.],
2022-05-18T03:51:55.1478037Z         [-1.,  1.],
2022-05-18T03:51:55.1478536Z         [-1., -1.],
2022-05-18T03:51:55.1479368Z         [-1., -1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([1., 1., 1., 1.])). Grads dict has 2 entries: {Parameter containing:
2022-05-18T03:51:55.1480233Z tensor([[-1.,  1.,  1.]], requires_grad=True): tensor([[0., 0., 0.]]), Parameter containing:
2022-05-18T03:51:55.1480790Z tensor([[-1.,  1.],
2022-05-18T03:51:55.1481232Z         [ 1.,  1.]], requires_grad=True): tensor([[0., 0.],
2022-05-18T03:51:55.1481644Z         [0., 0.]])}
2022-05-18T03:51:55.1488778Z 2022-05-18 03:51:55,148 ddp_under_dist_autograd_test.py:261 INFO p:process 3 t:Dummy-4: Loss is 0.0 for mini batch: FeatureSet(dense_features=tensor([[ 1.,  1.],
2022-05-18T03:51:55.1489350Z         [ 1.,  1.],
2022-05-18T03:51:55.1489725Z         [ 1., -1.],
2022-05-18T03:51:55.1490449Z         [ 1., -1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([1., 1., 1., 1.])). Grads dict has 2 entries: {Parameter containing:
2022-05-18T03:51:55.1491262Z tensor([[-1.,  1.,  1.]], requires_grad=True): tensor([[0., 0., 0.]]), Parameter containing:
2022-05-18T03:51:55.1491679Z tensor([[-1.,  1.],
2022-05-18T03:51:55.1492116Z         [ 1.,  1.]], requires_grad=True): tensor([[0., 0.],
2022-05-18T03:51:55.1492516Z         [0., 0.]])}
2022-05-18T03:51:55.1783327Z 2022-05-18 03:51:55,177 ddp_under_dist_autograd_test.py:261 INFO p:process 2 t:Dummy-5: Loss is 56.0 for mini batch: FeatureSet(dense_features=tensor([[-1.,  1.],
2022-05-18T03:51:55.1783978Z         [-1.,  1.],
2022-05-18T03:51:55.1784262Z         [-1., -1.],
2022-05-18T03:51:55.1784973Z         [-1., -1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([1., 1., 1., 1.])). Grads dict has 2 entries: {Parameter containing:
2022-05-18T03:51:55.1785827Z tensor([[-1.,  1.,  1.]], requires_grad=True): tensor([[0., 0., 0.]]), Parameter containing:
2022-05-18T03:51:55.1786246Z tensor([[-1.,  1.],
2022-05-18T03:51:55.1786561Z         [ 1.,  1.]], requires_grad=True): tensor([[0., 0.],
2022-05-18T03:51:55.1786882Z         [0., 0.]])}
2022-05-18T03:51:55.1787829Z 2022-05-18 03:51:55,177 ddp_under_dist_autograd_test.py:261 INFO p:process 1 t:Dummy-5: Loss is -56.0 for mini batch: FeatureSet(dense_features=tensor([[-1., -1.],
2022-05-18T03:51:55.1788492Z         [-1., -1.],
2022-05-18T03:51:55.1788907Z         [-1.,  1.],
2022-05-18T03:51:55.1789770Z         [-1.,  1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([-1., -1., -1., -1.])). Grads dict has 2 entries: {Parameter containing:
2022-05-18T03:51:55.1790695Z tensor([[-1.,  1.,  1.]], requires_grad=True): tensor([[0., 0., 0.]]), Parameter containing:
2022-05-18T03:51:55.1791243Z tensor([[-1.,  1.],
2022-05-18T03:51:55.1791696Z         [ 1.,  1.]], requires_grad=True): tensor([[0., 0.],
2022-05-18T03:51:55.1792126Z         [0., 0.]])}
2022-05-18T03:51:55.1817083Z 2022-05-18 03:51:55,181 ddp_under_dist_autograd_test.py:261 INFO p:process 0 t:Dummy-5: Loss is 0.0 for mini batch: FeatureSet(dense_features=tensor([[ 1., -1.],
2022-05-18T03:51:55.1817745Z         [ 1., -1.],
2022-05-18T03:51:55.1818109Z         [ 1.,  1.],
2022-05-18T03:51:55.1819065Z         [ 1.,  1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([-1., -1., -1., -1.])). Grads dict has 2 entries: {Parameter containing:
2022-05-18T03:51:55.1819967Z tensor([[-1.,  1.,  1.]], requires_grad=True): tensor([[0., 0., 0.]]), Parameter containing:
2022-05-18T03:51:55.1820541Z tensor([[-1.,  1.],
2022-05-18T03:51:55.1820999Z         [ 1.,  1.]], requires_grad=True): tensor([[0., 0.],
2022-05-18T03:51:55.1821411Z         [0., 0.]])}
2022-05-18T03:51:55.1826033Z 2022-05-18 03:51:55,177 ddp_under_dist_autograd_test.py:261 INFO p:process 3 t:Dummy-5: Loss is 0.0 for mini batch: FeatureSet(dense_features=tensor([[ 1.,  1.],
2022-05-18T03:51:55.1826650Z         [ 1.,  1.],
2022-05-18T03:51:55.1827089Z         [ 1., -1.],
2022-05-18T03:51:55.1828217Z         [ 1., -1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([1., 1., 1., 1.])). Grads dict has 2 entries: {Parameter containing:
2022-05-18T03:51:55.1829205Z tensor([[-1.,  1.,  1.]], requires_grad=True): tensor([[0., 0., 0.]]), Parameter containing:
2022-05-18T03:51:55.1829783Z tensor([[-1.,  1.],
2022-05-18T03:51:55.1830229Z         [ 1.,  1.]], requires_grad=True): tensor([[0., 0.],
2022-05-18T03:51:55.1830649Z         [0., 0.]])}
2022-05-18T03:51:55.1906761Z 2022-05-18 03:51:55,190 ddp_under_dist_autograd_test.py:364 INFO p:process 0 t:MainThread: Exiting the trainer #0...
2022-05-18T03:51:55.1915565Z 2022-05-18 03:51:55,191 ddp_under_dist_autograd_test.py:364 INFO p:process 1 t:MainThread: Exiting the trainer #1...
2022-05-18T03:51:55.1925611Z 2022-05-18 03:51:55,192 ddp_under_dist_autograd_test.py:364 INFO p:process 2 t:MainThread: Exiting the trainer #2...
2022-05-18T03:51:55.1933155Z 2022-05-18 03:51:55,192 ddp_under_dist_autograd_test.py:364 INFO p:process 3 t:MainThread: Exiting the trainer #3...
2022-05-18T03:51:55.1944312Z 2022-05-18 03:51:55,194 ddp_under_dist_autograd_test.py:344 INFO p:process 4 t:MainThread: Exiting remote worker.
2022-05-18T03:51:55.5193453Z ok (2.632s)
2022-05-18T03:51:55.5193604Z 
2022-05-18T03:51:55.5193907Z ----------------------------------------------------------------------
2022-05-18T03:51:55.5194173Z Ran 1 test in 2.633s
2022-05-18T03:51:55.5194287Z 
2022-05-18T03:51:55.5194336Z OK
2022-05-18T03:51:55.5194427Z 
2022-05-18T03:51:55.5194521Z Generating XML reports...
2022-05-18T03:51:55.5226852Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDdpUnderDistAutogradTest-20220518035152.xml
2022-05-18T03:51:56.2855322Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1o7ekr6w
2022-05-18T03:51:56.2855824Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1o7ekr6w/_remote_module_non_scriptable.py
2022-05-18T03:51:56.5378152Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T03:51:56.5387503Z 
2022-05-18T03:51:56.5387599Z Running tests...
2022-05-18T03:51:56.5387996Z ----------------------------------------------------------------------
2022-05-18T03:51:56.8505381Z   test_backward_ddp_outside_uneven_inputs (__main__.TensorPipeDdpUnderDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8376
2022-05-18T03:51:56.8527789Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8377
2022-05-18T03:51:56.8551371Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8378
2022-05-18T03:51:56.8576332Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8379
2022-05-18T03:51:56.8600925Z INFO:torch.testing._internal.common_distributed:Started process 4 with pid 8380
2022-05-18T03:51:56.8627063Z INFO:torch.testing._internal.common_distributed:Started process 5 with pid 8381
2022-05-18T03:51:57.6343559Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_v_yaqq2
2022-05-18T03:51:57.6345366Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_v_yaqq2/_remote_module_non_scriptable.py
2022-05-18T03:51:57.6398918Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcm5rf_7_
2022-05-18T03:51:57.6401432Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcm5rf_7_/_remote_module_non_scriptable.py
2022-05-18T03:51:57.6945776Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjqdfbbyz
2022-05-18T03:51:57.6946626Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjqdfbbyz/_remote_module_non_scriptable.py
2022-05-18T03:51:57.7081777Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8q6p1lrg
2022-05-18T03:51:57.7082558Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8q6p1lrg/_remote_module_non_scriptable.py
2022-05-18T03:51:57.8269435Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyacxhvjh
2022-05-18T03:51:57.8273725Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyacxhvjh/_remote_module_non_scriptable.py
2022-05-18T03:51:57.8723935Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphhsxtve9
2022-05-18T03:51:57.8725692Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphhsxtve9/_remote_module_non_scriptable.py
2022-05-18T03:51:57.9475324Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 5
2022-05-18T03:51:57.9583164Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:51:58.0335076Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:51:58.0357921Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:51:58.1560665Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 4
2022-05-18T03:51:58.1726450Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:51:58.4575727Z 2022-05-18 03:51:58,456 ddp_under_dist_autograd_test.py:348 INFO p:process 0 t:MainThread: Running the trainer #0...
2022-05-18T03:51:58.4576932Z 2022-05-18 03:51:58,456 ddp_under_dist_autograd_test.py:350 INFO p:process 0 t:MainThread: Initing trainer process group by trainer #0 with ranks [0, 1, 2, 3]
2022-05-18T03:51:58.4671103Z 2022-05-18 03:51:58,466 ddp_under_dist_autograd_test.py:348 INFO p:process 1 t:MainThread: Running the trainer #1...
2022-05-18T03:51:58.4672256Z 2022-05-18 03:51:58,466 ddp_under_dist_autograd_test.py:350 INFO p:process 1 t:MainThread: Initing trainer process group by trainer #1 with ranks [0, 1, 2, 3]
2022-05-18T03:51:58.4769829Z 2022-05-18 03:51:58,476 ddp_under_dist_autograd_test.py:348 INFO p:process 2 t:MainThread: Running the trainer #2...
2022-05-18T03:51:58.4771170Z 2022-05-18 03:51:58,476 ddp_under_dist_autograd_test.py:350 INFO p:process 2 t:MainThread: Initing trainer process group by trainer #2 with ranks [0, 1, 2, 3]
2022-05-18T03:51:58.4967277Z 2022-05-18 03:51:58,496 ddp_under_dist_autograd_test.py:329 INFO p:process 4 t:MainThread: The remote worker is running.
2022-05-18T03:51:58.4978733Z 2022-05-18 03:51:58,497 ddp_under_dist_autograd_test.py:368 INFO p:process 5 t:MainThread: Running the master process...
2022-05-18T03:51:58.4987260Z 2022-05-18 03:51:58,497 ddp_under_dist_autograd_test.py:348 INFO p:process 3 t:MainThread: Running the trainer #3...
2022-05-18T03:51:58.4988435Z 2022-05-18 03:51:58,497 ddp_under_dist_autograd_test.py:350 INFO p:process 3 t:MainThread: Initing trainer process group by trainer #3 with ranks [0, 1, 2, 3]
2022-05-18T03:51:58.5302461Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:51:58.5303588Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:51:58.5308187Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 4
2022-05-18T03:51:58.5313666Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:51:58.5409002Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:51:58.5420292Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 5
2022-05-18T03:51:58.5425053Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 6 nodes.
2022-05-18T03:51:58.5426240Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:1 with 6 nodes.
2022-05-18T03:51:58.5427759Z 2022-05-18 03:51:58,541 ddp_under_dist_autograd_test.py:359 INFO p:process 0 t:MainThread: Waiting for shutdown signal on trainer #0...
2022-05-18T03:51:58.5431458Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 4
2022-05-18T03:51:58.5434729Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 6 nodes.
2022-05-18T03:51:58.5445479Z 2022-05-18 03:51:58,543 ddp_under_dist_autograd_test.py:359 INFO p:process 3 t:MainThread: Waiting for shutdown signal on trainer #3...
2022-05-18T03:51:58.5446632Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:1 with 6 nodes.
2022-05-18T03:51:58.5450562Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 6 nodes.
2022-05-18T03:51:58.5451710Z 2022-05-18 03:51:58,544 ddp_under_dist_autograd_test.py:359 INFO p:process 2 t:MainThread: Waiting for shutdown signal on trainer #2...
2022-05-18T03:51:58.5464130Z 2022-05-18 03:51:58,545 ddp_under_dist_autograd_test.py:382 INFO p:process 5 t:MainThread: Created remote rrefs on master
2022-05-18T03:51:58.5465417Z 2022-05-18 03:51:58,545 ddp_under_dist_autograd_test.py:396 INFO p:process 5 t:MainThread: Running DDP + RPC test with simulating uneven inputs across trainers.
2022-05-18T03:51:58.5473153Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 5
2022-05-18T03:51:58.5507398Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 6 nodes.
2022-05-18T03:51:58.5510941Z 2022-05-18 03:51:58,550 ddp_under_dist_autograd_test.py:359 INFO p:process 1 t:MainThread: Waiting for shutdown signal on trainer #1...
2022-05-18T03:51:58.5775443Z 2022-05-18 03:51:58,576 ddp_under_dist_autograd_test.py:94 INFO p:process 4 t:Dummy-2: Initing RemoteEM with 2 3
2022-05-18T03:51:58.5784835Z 2022-05-18 03:51:58,577 ddp_under_dist_autograd_test.py:120 INFO p:process 4 t:Dummy-3: Initing RemoteNet with 5 3
2022-05-18T03:51:58.7218991Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T03:51:58.7328038Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T03:51:58.7330908Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 3
2022-05-18T03:51:58.7335825Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-05-18T03:51:58.7337170Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 6 nodes.
2022-05-18T03:51:58.7342307Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 6 nodes.
2022-05-18T03:51:58.7386407Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:2 with 6 nodes.
2022-05-18T03:51:58.7421518Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:2 with 6 nodes.
2022-05-18T03:51:58.7425703Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 6 nodes.
2022-05-18T03:51:58.7435538Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:2 with 6 nodes.
2022-05-18T03:51:58.7448816Z 2022-05-18 03:51:58,744 ddp_under_dist_autograd_test.py:159 INFO p:process 3 t:Dummy-2: HybridModel has 2 groups of parameters.
2022-05-18T03:51:58.7449901Z 2022-05-18 03:51:58,744 ddp_under_dist_autograd_test.py:202 INFO p:process 3 t:Dummy-2: Wrapping the whole hybrid module into DDP.
2022-05-18T03:51:58.7464690Z 2022-05-18 03:51:58,746 ddp_under_dist_autograd_test.py:159 INFO p:process 1 t:Dummy-2: HybridModel has 2 groups of parameters.
2022-05-18T03:51:58.7472232Z 2022-05-18 03:51:58,746 ddp_under_dist_autograd_test.py:202 INFO p:process 1 t:Dummy-2: Wrapping the whole hybrid module into DDP.
2022-05-18T03:51:58.7481131Z 2022-05-18 03:51:58,747 ddp_under_dist_autograd_test.py:159 INFO p:process 0 t:Dummy-2: HybridModel has 2 groups of parameters.
2022-05-18T03:51:58.7504320Z 2022-05-18 03:51:58,749 ddp_under_dist_autograd_test.py:159 INFO p:process 2 t:Dummy-2: HybridModel has 2 groups of parameters.
2022-05-18T03:51:58.7505430Z 2022-05-18 03:51:58,749 ddp_under_dist_autograd_test.py:202 INFO p:process 2 t:Dummy-2: Wrapping the whole hybrid module into DDP.
2022-05-18T03:51:58.7511768Z 2022-05-18 03:51:58,750 ddp_under_dist_autograd_test.py:202 INFO p:process 0 t:Dummy-2: Wrapping the whole hybrid module into DDP.
2022-05-18T03:51:58.7826079Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:621: UserWarning: The `check_reduction` argument in `DistributedDataParallel` module is deprecated. Please avoid using it.
2022-05-18T03:51:58.7827094Z   "The `check_reduction` argument in `DistributedDataParallel` "
2022-05-18T03:51:58.7828369Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:621: UserWarning: The `check_reduction` argument in `DistributedDataParallel` module is deprecated. Please avoid using it.
2022-05-18T03:51:58.7829352Z   "The `check_reduction` argument in `DistributedDataParallel` "
2022-05-18T03:51:58.7915126Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:621: UserWarning: The `check_reduction` argument in `DistributedDataParallel` module is deprecated. Please avoid using it.
2022-05-18T03:51:58.7916144Z   "The `check_reduction` argument in `DistributedDataParallel` "
2022-05-18T03:51:58.7954584Z /opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py:621: UserWarning: The `check_reduction` argument in `DistributedDataParallel` module is deprecated. Please avoid using it.
2022-05-18T03:51:58.7955555Z   "The `check_reduction` argument in `DistributedDataParallel` "
2022-05-18T03:51:58.7981910Z 2022-05-18 03:51:58,797 ddp_under_dist_autograd_test.py:211 INFO p:process 3 t:Dummy-2: Succeeded in creating a HybridModel instance with 2 ddp params and 0 other local params.
2022-05-18T03:51:58.7983899Z 2022-05-18 03:51:58,797 ddp_under_dist_autograd_test.py:211 INFO p:process 1 t:Dummy-2: Succeeded in creating a HybridModel instance with 2 ddp params and 0 other local params.
2022-05-18T03:51:58.7985595Z 2022-05-18 03:51:58,797 ddp_under_dist_autograd_test.py:211 INFO p:process 2 t:Dummy-2: Succeeded in creating a HybridModel instance with 2 ddp params and 0 other local params.
2022-05-18T03:51:58.7987052Z 2022-05-18 03:51:58,798 ddp_under_dist_autograd_test.py:248 INFO p:process 1 t:Dummy-3: Trainer reduced input patches from 2
2022-05-18T03:51:58.7987865Z                     to 1 to simulate uneven inputs.
2022-05-18T03:51:58.8003270Z 2022-05-18 03:51:58,799 ddp_under_dist_autograd_test.py:211 INFO p:process 0 t:Dummy-2: Succeeded in creating a HybridModel instance with 2 ddp params and 0 other local params.
2022-05-18T03:51:58.8010539Z 2022-05-18 03:51:58,800 ddp_under_dist_autograd_test.py:248 INFO p:process 0 t:Dummy-3: Trainer reduced input patches from 2
2022-05-18T03:51:58.8011228Z                     to 1 to simulate uneven inputs.
2022-05-18T03:51:58.8501889Z 2022-05-18 03:51:58,849 ddp_under_dist_autograd_test.py:261 INFO p:process 1 t:Dummy-3: Loss is 0.0 for mini batch: FeatureSet(dense_features=tensor([[-1., -1.],
2022-05-18T03:51:58.8502660Z         [-1., -1.],
2022-05-18T03:51:58.8503170Z         [-1.,  1.],
2022-05-18T03:51:58.8504007Z         [-1.,  1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([-1., -1., -1., -1.])). Grads dict has 2 entries: {Parameter containing:
2022-05-18T03:51:58.8504932Z tensor([[-1.,  1.,  1.]], requires_grad=True): tensor([[ 0., 14., 14.]]), Parameter containing:
2022-05-18T03:51:58.8505337Z tensor([[-1.,  1.],
2022-05-18T03:51:58.8505664Z         [ 1.,  1.]], requires_grad=True): tensor([[ 0., 12.],
2022-05-18T03:51:58.8505970Z         [ 0.,  4.]])}
2022-05-18T03:51:58.8506789Z 2022-05-18 03:51:58,849 ddp_under_dist_autograd_test.py:261 INFO p:process 0 t:Dummy-3: Loss is 28.0 for mini batch: FeatureSet(dense_features=tensor([[ 1., -1.],
2022-05-18T03:51:58.8507885Z         [ 1., -1.],
2022-05-18T03:51:58.8508216Z         [ 1.,  1.],
2022-05-18T03:51:58.8509162Z         [ 1.,  1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([-1., -1., -1., -1.])). Grads dict has 2 entries: {Parameter containing:
2022-05-18T03:51:58.8510082Z tensor([[-1.,  1.,  1.]], requires_grad=True): tensor([[ 0., 14., 14.]]), Parameter containing:
2022-05-18T03:51:58.8510637Z tensor([[-1.,  1.],
2022-05-18T03:51:58.8511068Z         [ 1.,  1.]], requires_grad=True): tensor([[ 0., 12.],
2022-05-18T03:51:58.8511488Z         [ 0.,  4.]])}
2022-05-18T03:51:58.8512415Z 2022-05-18 03:51:58,849 ddp_under_dist_autograd_test.py:261 INFO p:process 3 t:Dummy-3: Loss is 28.0 for mini batch: FeatureSet(dense_features=tensor([[ 1.,  1.],
2022-05-18T03:51:58.8513013Z         [ 1.,  1.],
2022-05-18T03:51:58.8513418Z         [ 1., -1.],
2022-05-18T03:51:58.8514240Z         [ 1., -1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([1., 1., 1., 1.])). Grads dict has 2 entries: {Parameter containing:
2022-05-18T03:51:58.8515124Z tensor([[-1.,  1.,  1.]], requires_grad=True): tensor([[ 0., 14., 14.]]), Parameter containing:
2022-05-18T03:51:58.8515681Z tensor([[-1.,  1.],
2022-05-18T03:51:58.8516127Z         [ 1.,  1.]], requires_grad=True): tensor([[ 0., 12.],
2022-05-18T03:51:58.8516552Z         [ 0.,  4.]])}
2022-05-18T03:51:58.8530738Z 2022-05-18 03:51:58,852 ddp_under_dist_autograd_test.py:261 INFO p:process 2 t:Dummy-3: Loss is 56.0 for mini batch: FeatureSet(dense_features=tensor([[-1.,  1.],
2022-05-18T03:51:58.8531436Z         [-1.,  1.],
2022-05-18T03:51:58.8531842Z         [-1., -1.],
2022-05-18T03:51:58.8532653Z         [-1., -1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([1., 1., 1., 1.])). Grads dict has 2 entries: {Parameter containing:
2022-05-18T03:51:58.8533555Z tensor([[-1.,  1.,  1.]], requires_grad=True): tensor([[ 0., 14., 14.]]), Parameter containing:
2022-05-18T03:51:58.8534108Z tensor([[-1.,  1.],
2022-05-18T03:51:58.8534573Z         [ 1.,  1.]], requires_grad=True): tensor([[ 0., 12.],
2022-05-18T03:51:58.8534971Z         [ 0.,  4.]])}
2022-05-18T03:51:58.8543485Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T03:51:58.8544299Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T03:51:58.8673279Z 2022-05-18 03:51:58,866 ddp_under_dist_autograd_test.py:261 INFO p:process 2 t:Dummy-3: Loss is 0.0 for mini batch: FeatureSet(dense_features=tensor([[-1.,  1.],
2022-05-18T03:51:58.8674019Z         [-1.,  1.],
2022-05-18T03:51:58.8674444Z         [-1., -1.],
2022-05-18T03:51:58.8675278Z         [-1., -1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([1., 1., 1., 1.])). Grads dict has 2 entries: {Parameter containing:
2022-05-18T03:51:58.8676166Z tensor([[-1.,  1.,  1.]], requires_grad=True): tensor([[7., 0., 0.]]), Parameter containing:
2022-05-18T03:51:58.8676734Z tensor([[-1.,  1.],
2022-05-18T03:51:58.8677305Z         [ 1.,  1.]], requires_grad=True): tensor([[ 2., -2.],
2022-05-18T03:51:58.8677784Z         [-2.,  2.]])}
2022-05-18T03:51:58.8678750Z 2022-05-18 03:51:58,866 ddp_under_dist_autograd_test.py:261 INFO p:process 3 t:Dummy-3: Loss is -28.0 for mini batch: FeatureSet(dense_features=tensor([[ 1.,  1.],
2022-05-18T03:51:58.8679374Z         [ 1.,  1.],
2022-05-18T03:51:58.8679779Z         [ 1., -1.],
2022-05-18T03:51:58.8680578Z         [ 1., -1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([1., 1., 1., 1.])). Grads dict has 2 entries: {Parameter containing:
2022-05-18T03:51:58.8681458Z tensor([[-1.,  1.,  1.]], requires_grad=True): tensor([[7., 0., 0.]]), Parameter containing:
2022-05-18T03:51:58.8682013Z tensor([[-1.,  1.],
2022-05-18T03:51:58.8682560Z         [ 1.,  1.]], requires_grad=True): tensor([[ 2., -2.],
2022-05-18T03:51:58.8683040Z         [-2.,  2.]])}
2022-05-18T03:51:58.8716252Z 2022-05-18 03:51:58,871 ddp_under_dist_autograd_test.py:248 INFO p:process 1 t:Dummy-4: Trainer reduced input patches from 2
2022-05-18T03:51:58.8717343Z                     to 1 to simulate uneven inputs.
2022-05-18T03:51:58.8718924Z 2022-05-18 03:51:58,871 ddp_under_dist_autograd_test.py:248 INFO p:process 0 t:Dummy-4: Trainer reduced input patches from 2
2022-05-18T03:51:58.8719518Z                     to 1 to simulate uneven inputs.
2022-05-18T03:51:58.8916025Z 2022-05-18 03:51:58,890 ddp_under_dist_autograd_test.py:261 INFO p:process 1 t:Dummy-4: Loss is 0.0 for mini batch: FeatureSet(dense_features=tensor([[-1., -1.],
2022-05-18T03:51:58.8916636Z         [-1., -1.],
2022-05-18T03:51:58.8916998Z         [-1.,  1.],
2022-05-18T03:51:58.8917738Z         [-1.,  1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([-1., -1., -1., -1.])). Grads dict has 2 entries: {Parameter containing:
2022-05-18T03:51:58.8918521Z tensor([[-1.,  1.,  1.]], requires_grad=True): tensor([[ 0., 14., 14.]]), Parameter containing:
2022-05-18T03:51:58.8919024Z tensor([[-1.,  1.],
2022-05-18T03:51:58.8919483Z         [ 1.,  1.]], requires_grad=True): tensor([[ 0., 12.],
2022-05-18T03:51:58.8919924Z         [ 0.,  4.]])}
2022-05-18T03:51:58.8949140Z 2022-05-18 03:51:58,894 ddp_under_dist_autograd_test.py:261 INFO p:process 2 t:Dummy-4: Loss is 56.0 for mini batch: FeatureSet(dense_features=tensor([[-1.,  1.],
2022-05-18T03:51:58.8949729Z         [-1.,  1.],
2022-05-18T03:51:58.8950100Z         [-1., -1.],
2022-05-18T03:51:58.8950827Z         [-1., -1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([1., 1., 1., 1.])). Grads dict has 2 entries: {Parameter containing:
2022-05-18T03:51:58.8951688Z tensor([[-1.,  1.,  1.]], requires_grad=True): tensor([[ 0., 14., 14.]]), Parameter containing:
2022-05-18T03:51:58.8952242Z tensor([[-1.,  1.],
2022-05-18T03:51:58.8952695Z         [ 1.,  1.]], requires_grad=True): tensor([[ 0., 12.],
2022-05-18T03:51:58.8953121Z         [ 0.,  4.]])}
2022-05-18T03:51:58.8954045Z 2022-05-18 03:51:58,894 ddp_under_dist_autograd_test.py:261 INFO p:process 0 t:Dummy-4: Loss is 28.0 for mini batch: FeatureSet(dense_features=tensor([[ 1., -1.],
2022-05-18T03:51:58.8954726Z         [ 1., -1.],
2022-05-18T03:51:58.8955069Z         [ 1.,  1.],
2022-05-18T03:51:58.8955907Z         [ 1.,  1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([-1., -1., -1., -1.])). Grads dict has 2 entries: {Parameter containing:
2022-05-18T03:51:58.8956804Z tensor([[-1.,  1.,  1.]], requires_grad=True): tensor([[ 0., 14., 14.]]), Parameter containing:
2022-05-18T03:51:58.8957364Z tensor([[-1.,  1.],
2022-05-18T03:51:58.8957808Z         [ 1.,  1.]], requires_grad=True): tensor([[ 0., 12.],
2022-05-18T03:51:58.8958206Z         [ 0.,  4.]])}
2022-05-18T03:51:58.8959135Z 2022-05-18 03:51:58,891 ddp_under_dist_autograd_test.py:261 INFO p:process 3 t:Dummy-4: Loss is 28.0 for mini batch: FeatureSet(dense_features=tensor([[ 1.,  1.],
2022-05-18T03:51:58.8959747Z         [ 1.,  1.],
2022-05-18T03:51:58.8960127Z         [ 1., -1.],
2022-05-18T03:51:58.8960946Z         [ 1., -1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([1., 1., 1., 1.])). Grads dict has 2 entries: {Parameter containing:
2022-05-18T03:51:58.8961830Z tensor([[-1.,  1.,  1.]], requires_grad=True): tensor([[ 0., 14., 14.]]), Parameter containing:
2022-05-18T03:51:58.8962372Z tensor([[-1.,  1.],
2022-05-18T03:51:58.8962817Z         [ 1.,  1.]], requires_grad=True): tensor([[ 0., 12.],
2022-05-18T03:51:58.8963234Z         [ 0.,  4.]])}
2022-05-18T03:51:58.9086718Z 2022-05-18 03:51:58,908 ddp_under_dist_autograd_test.py:261 INFO p:process 2 t:Dummy-4: Loss is 0.0 for mini batch: FeatureSet(dense_features=tensor([[-1.,  1.],
2022-05-18T03:51:58.9087333Z         [-1.,  1.],
2022-05-18T03:51:58.9087694Z         [-1., -1.],
2022-05-18T03:51:58.9088417Z         [-1., -1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([1., 1., 1., 1.])). Grads dict has 2 entries: {Parameter containing:
2022-05-18T03:51:58.9089291Z tensor([[-1.,  1.,  1.]], requires_grad=True): tensor([[7., 0., 0.]]), Parameter containing:
2022-05-18T03:51:58.9089854Z tensor([[-1.,  1.],
2022-05-18T03:51:58.9090798Z         [ 1.,  1.]], requires_grad=True): tensor([[ 2., -2.],
2022-05-18T03:51:58.9091290Z         [-2.,  2.]])}
2022-05-18T03:51:58.9097918Z 2022-05-18 03:51:58,908 ddp_under_dist_autograd_test.py:261 INFO p:process 3 t:Dummy-4: Loss is -28.0 for mini batch: FeatureSet(dense_features=tensor([[ 1.,  1.],
2022-05-18T03:51:58.9098459Z         [ 1.,  1.],
2022-05-18T03:51:58.9098905Z         [ 1., -1.],
2022-05-18T03:51:58.9099595Z         [ 1., -1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([1., 1., 1., 1.])). Grads dict has 2 entries: {Parameter containing:
2022-05-18T03:51:58.9100358Z tensor([[-1.,  1.,  1.]], requires_grad=True): tensor([[7., 0., 0.]]), Parameter containing:
2022-05-18T03:51:58.9100860Z tensor([[-1.,  1.],
2022-05-18T03:51:58.9101453Z         [ 1.,  1.]], requires_grad=True): tensor([[ 2., -2.],
2022-05-18T03:51:58.9101927Z         [-2.,  2.]])}
2022-05-18T03:51:58.9148849Z 2022-05-18 03:51:58,914 ddp_under_dist_autograd_test.py:248 INFO p:process 0 t:Dummy-5: Trainer reduced input patches from 2
2022-05-18T03:51:58.9149446Z                     to 1 to simulate uneven inputs.
2022-05-18T03:51:58.9150211Z 2022-05-18 03:51:58,914 ddp_under_dist_autograd_test.py:248 INFO p:process 1 t:Dummy-5: Trainer reduced input patches from 2
2022-05-18T03:51:58.9150767Z                     to 1 to simulate uneven inputs.
2022-05-18T03:51:58.9354576Z 2022-05-18 03:51:58,934 ddp_under_dist_autograd_test.py:261 INFO p:process 1 t:Dummy-5: Loss is 0.0 for mini batch: FeatureSet(dense_features=tensor([[-1., -1.],
2022-05-18T03:51:58.9355232Z         [-1., -1.],
2022-05-18T03:51:58.9355572Z         [-1.,  1.],
2022-05-18T03:51:58.9356304Z         [-1.,  1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([-1., -1., -1., -1.])). Grads dict has 2 entries: {Parameter containing:
2022-05-18T03:51:58.9357085Z tensor([[-1.,  1.,  1.]], requires_grad=True): tensor([[ 0., 14., 14.]]), Parameter containing:
2022-05-18T03:51:58.9357540Z tensor([[-1.,  1.],
2022-05-18T03:51:58.9357908Z         [ 1.,  1.]], requires_grad=True): tensor([[ 0., 12.],
2022-05-18T03:51:58.9358278Z         [ 0.,  4.]])}
2022-05-18T03:51:58.9359092Z 2022-05-18 03:51:58,935 ddp_under_dist_autograd_test.py:261 INFO p:process 2 t:Dummy-5: Loss is 56.0 for mini batch: FeatureSet(dense_features=tensor([[-1.,  1.],
2022-05-18T03:51:58.9359661Z         [-1.,  1.],
2022-05-18T03:51:58.9360003Z         [-1., -1.],
2022-05-18T03:51:58.9360809Z         [-1., -1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([1., 1., 1., 1.])). Grads dict has 2 entries: {Parameter containing:
2022-05-18T03:51:58.9361687Z tensor([[-1.,  1.,  1.]], requires_grad=True): tensor([[ 0., 14., 14.]]), Parameter containing:
2022-05-18T03:51:58.9362249Z tensor([[-1.,  1.],
2022-05-18T03:51:58.9362692Z         [ 1.,  1.]], requires_grad=True): tensor([[ 0., 12.],
2022-05-18T03:51:58.9363110Z         [ 0.,  4.]])}
2022-05-18T03:51:58.9381075Z 2022-05-18 03:51:58,937 ddp_under_dist_autograd_test.py:261 INFO p:process 0 t:Dummy-5: Loss is 28.0 for mini batch: FeatureSet(dense_features=tensor([[ 1., -1.],
2022-05-18T03:51:58.9381687Z         [ 1., -1.],
2022-05-18T03:51:58.9383188Z         [ 1.,  1.],
2022-05-18T03:51:58.9384850Z         [ 1.,  1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([-1., -1., -1., -1.])). Grads dict has 2 entries: {Parameter containing:
2022-05-18T03:51:58.9385282Z tensor([[-1.,  1.,  1.]], requires_grad=True): tensor([[ 0., 14., 14.]]), Parameter containing:
2022-05-18T03:51:58.9385556Z tensor([[-1.,  1.],
2022-05-18T03:51:58.9385758Z         [ 1.,  1.]], requires_grad=True): tensor([[ 0., 12.],
2022-05-18T03:51:58.9385958Z         [ 0.,  4.]])}
2022-05-18T03:51:58.9402308Z 2022-05-18 03:51:58,937 ddp_under_dist_autograd_test.py:261 INFO p:process 3 t:Dummy-5: Loss is 28.0 for mini batch: FeatureSet(dense_features=tensor([[ 1.,  1.],
2022-05-18T03:51:58.9402862Z         [ 1.,  1.],
2022-05-18T03:51:58.9403263Z         [ 1., -1.],
2022-05-18T03:51:58.9404102Z         [ 1., -1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([1., 1., 1., 1.])). Grads dict has 2 entries: {Parameter containing:
2022-05-18T03:51:58.9405320Z tensor([[-1.,  1.,  1.]], requires_grad=True): tensor([[ 0., 14., 14.]]), Parameter containing:
2022-05-18T03:51:58.9405969Z tensor([[-1.,  1.],
2022-05-18T03:51:58.9406407Z         [ 1.,  1.]], requires_grad=True): tensor([[ 0., 12.],
2022-05-18T03:51:58.9406839Z         [ 0.,  4.]])}
2022-05-18T03:51:58.9544422Z 2022-05-18 03:51:58,953 ddp_under_dist_autograd_test.py:261 INFO p:process 3 t:Dummy-5: Loss is -28.0 for mini batch: FeatureSet(dense_features=tensor([[ 1.,  1.],
2022-05-18T03:51:58.9546022Z         [ 1.,  1.],
2022-05-18T03:51:58.9546378Z         [ 1., -1.],
2022-05-18T03:51:58.9547087Z         [ 1., -1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([1., 1., 1., 1.])). Grads dict has 2 entries: {Parameter containing:
2022-05-18T03:51:58.9547828Z tensor([[-1.,  1.,  1.]], requires_grad=True): tensor([[7., 0., 0.]]), Parameter containing:
2022-05-18T03:51:58.9548297Z tensor([[-1.,  1.],
2022-05-18T03:51:58.9548792Z         [ 1.,  1.]], requires_grad=True): tensor([[ 2., -2.],
2022-05-18T03:51:58.9549205Z         [-2.,  2.]])}
2022-05-18T03:51:58.9550101Z 2022-05-18 03:51:58,954 ddp_under_dist_autograd_test.py:261 INFO p:process 2 t:Dummy-5: Loss is 0.0 for mini batch: FeatureSet(dense_features=tensor([[-1.,  1.],
2022-05-18T03:51:58.9550777Z         [-1.,  1.],
2022-05-18T03:51:58.9551180Z         [-1., -1.],
2022-05-18T03:51:58.9551995Z         [-1., -1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([1., 1., 1., 1.])). Grads dict has 2 entries: {Parameter containing:
2022-05-18T03:51:58.9552862Z tensor([[-1.,  1.,  1.]], requires_grad=True): tensor([[7., 0., 0.]]), Parameter containing:
2022-05-18T03:51:58.9553420Z tensor([[-1.,  1.],
2022-05-18T03:51:58.9553981Z         [ 1.,  1.]], requires_grad=True): tensor([[ 2., -2.],
2022-05-18T03:51:58.9554441Z         [-2.,  2.]])}
2022-05-18T03:51:58.9601373Z 2022-05-18 03:51:58,959 ddp_under_dist_autograd_test.py:364 INFO p:process 0 t:MainThread: Exiting the trainer #0...
2022-05-18T03:51:58.9613117Z 2022-05-18 03:51:58,960 ddp_under_dist_autograd_test.py:364 INFO p:process 1 t:MainThread: Exiting the trainer #1...
2022-05-18T03:51:58.9619168Z 2022-05-18 03:51:58,961 ddp_under_dist_autograd_test.py:364 INFO p:process 2 t:MainThread: Exiting the trainer #2...
2022-05-18T03:51:58.9628108Z 2022-05-18 03:51:58,962 ddp_under_dist_autograd_test.py:364 INFO p:process 3 t:MainThread: Exiting the trainer #3...
2022-05-18T03:51:58.9641112Z 2022-05-18 03:51:58,963 ddp_under_dist_autograd_test.py:344 INFO p:process 4 t:MainThread: Exiting remote worker.
2022-05-18T03:51:59.0486569Z /opt/conda/lib/python3.7/tempfile.py:798: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmpcm5rf_7_'>
2022-05-18T03:51:59.0487377Z   _warnings.warn(warn_message, ResourceWarning)
2022-05-18T03:51:59.0549503Z /opt/conda/lib/python3.7/tempfile.py:798: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmp_v_yaqq2'>
2022-05-18T03:51:59.0549946Z   _warnings.warn(warn_message, ResourceWarning)
2022-05-18T03:51:59.0563827Z /opt/conda/lib/python3.7/tempfile.py:798: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmp8q6p1lrg'>
2022-05-18T03:51:59.0564501Z   _warnings.warn(warn_message, ResourceWarning)
2022-05-18T03:51:59.0577092Z /opt/conda/lib/python3.7/tempfile.py:798: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmphhsxtve9'>
2022-05-18T03:51:59.0577741Z   _warnings.warn(warn_message, ResourceWarning)
2022-05-18T03:51:59.2690932Z ok (2.730s)
2022-05-18T03:51:59.2691139Z 
2022-05-18T03:51:59.2691501Z ----------------------------------------------------------------------
2022-05-18T03:51:59.2691842Z Ran 1 test in 2.730s
2022-05-18T03:51:59.2691996Z 
2022-05-18T03:51:59.2692051Z OK
2022-05-18T03:51:59.2692142Z 
2022-05-18T03:51:59.2692239Z Generating XML reports...
2022-05-18T03:51:59.2727157Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDdpUnderDistAutogradTest-20220518035156.xml
2022-05-18T03:52:00.0366276Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1g8fwi_0
2022-05-18T03:52:00.0367641Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1g8fwi_0/_remote_module_non_scriptable.py
2022-05-18T03:52:00.2894886Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T03:52:00.2905094Z 
2022-05-18T03:52:00.2905387Z Running tests...
2022-05-18T03:52:00.2906022Z ----------------------------------------------------------------------
2022-05-18T03:52:00.6030385Z   test_backward_no_ddp (__main__.TensorPipeDdpUnderDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8793
2022-05-18T03:52:00.6053255Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8794
2022-05-18T03:52:00.6075831Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8795
2022-05-18T03:52:00.6100099Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8796
2022-05-18T03:52:00.6131465Z INFO:torch.testing._internal.common_distributed:Started process 4 with pid 8797
2022-05-18T03:52:00.6162351Z INFO:torch.testing._internal.common_distributed:Started process 5 with pid 8798
2022-05-18T03:52:01.4341854Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj_86ro39
2022-05-18T03:52:01.4344186Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj_86ro39/_remote_module_non_scriptable.py
2022-05-18T03:52:01.4533935Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3270jv5l
2022-05-18T03:52:01.4536162Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3270jv5l/_remote_module_non_scriptable.py
2022-05-18T03:52:01.4770375Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppvcaf982
2022-05-18T03:52:01.4771998Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppvcaf982/_remote_module_non_scriptable.py
2022-05-18T03:52:01.5835925Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu3mjnnfx
2022-05-18T03:52:01.5837036Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu3mjnnfx/_remote_module_non_scriptable.py
2022-05-18T03:52:01.6054864Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkk5cbm0n
2022-05-18T03:52:01.6059732Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkk5cbm0n/_remote_module_non_scriptable.py
2022-05-18T03:52:01.6060501Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgv2f_oqw
2022-05-18T03:52:01.6061235Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgv2f_oqw/_remote_module_non_scriptable.py
2022-05-18T03:52:01.6851650Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:52:01.7057169Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 4
2022-05-18T03:52:01.8473110Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:52:01.9244009Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:52:01.9281052Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 5
2022-05-18T03:52:01.9430960Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:52:02.2323268Z 2022-05-18 03:52:02,231 ddp_under_dist_autograd_test.py:348 INFO p:process 0 t:MainThread: Running the trainer #0...
2022-05-18T03:52:02.2329616Z 2022-05-18 03:52:02,231 ddp_under_dist_autograd_test.py:350 INFO p:process 0 t:MainThread: Initing trainer process group by trainer #0 with ranks [0, 1, 2, 3]
2022-05-18T03:52:02.2421875Z 2022-05-18 03:52:02,241 ddp_under_dist_autograd_test.py:348 INFO p:process 1 t:MainThread: Running the trainer #1...
2022-05-18T03:52:02.2428515Z 2022-05-18 03:52:02,241 ddp_under_dist_autograd_test.py:350 INFO p:process 1 t:MainThread: Initing trainer process group by trainer #1 with ranks [0, 1, 2, 3]
2022-05-18T03:52:02.2523118Z 2022-05-18 03:52:02,251 ddp_under_dist_autograd_test.py:348 INFO p:process 2 t:MainThread: Running the trainer #2...
2022-05-18T03:52:02.2529091Z 2022-05-18 03:52:02,251 ddp_under_dist_autograd_test.py:350 INFO p:process 2 t:MainThread: Initing trainer process group by trainer #2 with ranks [0, 1, 2, 3]
2022-05-18T03:52:02.2623544Z 2022-05-18 03:52:02,261 ddp_under_dist_autograd_test.py:348 INFO p:process 3 t:MainThread: Running the trainer #3...
2022-05-18T03:52:02.2624752Z 2022-05-18 03:52:02,261 ddp_under_dist_autograd_test.py:350 INFO p:process 3 t:MainThread: Initing trainer process group by trainer #3 with ranks [0, 1, 2, 3]
2022-05-18T03:52:02.2721340Z 2022-05-18 03:52:02,271 ddp_under_dist_autograd_test.py:329 INFO p:process 4 t:MainThread: The remote worker is running.
2022-05-18T03:52:02.2723363Z 2022-05-18 03:52:02,271 ddp_under_dist_autograd_test.py:368 INFO p:process 5 t:MainThread: Running the master process...
2022-05-18T03:52:02.2941895Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:52:02.2951128Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:52:02.2958968Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:52:02.2960047Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:52:02.3058813Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 5
2022-05-18T03:52:02.3067724Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 4
2022-05-18T03:52:02.3070092Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:1 with 6 nodes.
2022-05-18T03:52:02.3071265Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 6 nodes.
2022-05-18T03:52:02.3154772Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 6 nodes.
2022-05-18T03:52:02.3161778Z 2022-05-18 03:52:02,315 ddp_under_dist_autograd_test.py:359 INFO p:process 0 t:MainThread: Waiting for shutdown signal on trainer #0...
2022-05-18T03:52:02.3165676Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:1 with 6 nodes.
2022-05-18T03:52:02.3175012Z 2022-05-18 03:52:02,317 ddp_under_dist_autograd_test.py:382 INFO p:process 5 t:MainThread: Created remote rrefs on master
2022-05-18T03:52:02.3183227Z 2022-05-18 03:52:02,317 ddp_under_dist_autograd_test.py:359 INFO p:process 2 t:MainThread: Waiting for shutdown signal on trainer #2...
2022-05-18T03:52:02.3189727Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 6 nodes.
2022-05-18T03:52:02.3194622Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 6 nodes.
2022-05-18T03:52:02.3204694Z 2022-05-18 03:52:02,319 ddp_under_dist_autograd_test.py:359 INFO p:process 3 t:MainThread: Waiting for shutdown signal on trainer #3...
2022-05-18T03:52:02.3209718Z 2022-05-18 03:52:02,319 ddp_under_dist_autograd_test.py:359 INFO p:process 1 t:MainThread: Waiting for shutdown signal on trainer #1...
2022-05-18T03:52:02.3282291Z 2022-05-18 03:52:02,327 ddp_under_dist_autograd_test.py:159 INFO p:process 0 t:Dummy-2: HybridModel has 2 groups of parameters.
2022-05-18T03:52:02.3285339Z 2022-05-18 03:52:02,327 ddp_under_dist_autograd_test.py:211 INFO p:process 0 t:Dummy-2: Succeeded in creating a HybridModel instance with 0 ddp params and 2 other local params.
2022-05-18T03:52:02.3415901Z 2022-05-18 03:52:02,340 ddp_under_dist_autograd_test.py:94 INFO p:process 4 t:Dummy-2: Initing RemoteEM with 2 3
2022-05-18T03:52:02.3419727Z 2022-05-18 03:52:02,341 ddp_under_dist_autograd_test.py:120 INFO p:process 4 t:Dummy-3: Initing RemoteNet with 5 3
2022-05-18T03:52:02.3668722Z 2022-05-18 03:52:02,366 ddp_under_dist_autograd_test.py:261 INFO p:process 0 t:Dummy-3: Loss is 0.0 for mini batch: FeatureSet(dense_features=tensor([[ 1., -1.],
2022-05-18T03:52:02.3669404Z         [ 1., -1.],
2022-05-18T03:52:02.3669844Z         [ 1.,  1.],
2022-05-18T03:52:02.3670666Z         [ 1.,  1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([-1., -1., -1., -1.])). Grads dict has 2 entries: {Parameter containing:
2022-05-18T03:52:02.3671429Z tensor([[-1.,  1.,  1.]], requires_grad=True): tensor([[-56., -28., -28.]]), Parameter containing:
2022-05-18T03:52:02.3671780Z tensor([[-1.,  1.],
2022-05-18T03:52:02.3672182Z         [ 1.,  1.]], requires_grad=True): tensor([[-32., -16.],
2022-05-18T03:52:02.3672413Z         [  0., -16.]])}
2022-05-18T03:52:02.4340804Z 2022-05-18 03:52:02,433 ddp_under_dist_autograd_test.py:159 INFO p:process 1 t:Dummy-2: HybridModel has 2 groups of parameters.
2022-05-18T03:52:02.4341969Z 2022-05-18 03:52:02,433 ddp_under_dist_autograd_test.py:159 INFO p:process 3 t:Dummy-2: HybridModel has 2 groups of parameters.
2022-05-18T03:52:02.4343335Z 2022-05-18 03:52:02,433 ddp_under_dist_autograd_test.py:211 INFO p:process 1 t:Dummy-2: Succeeded in creating a HybridModel instance with 0 ddp params and 2 other local params.
2022-05-18T03:52:02.4344689Z 2022-05-18 03:52:02,433 ddp_under_dist_autograd_test.py:211 INFO p:process 3 t:Dummy-2: Succeeded in creating a HybridModel instance with 0 ddp params and 2 other local params.
2022-05-18T03:52:02.4602184Z 2022-05-18 03:52:02,459 ddp_under_dist_autograd_test.py:261 INFO p:process 1 t:Dummy-3: Loss is -56.0 for mini batch: FeatureSet(dense_features=tensor([[-1., -1.],
2022-05-18T03:52:02.4602937Z         [-1., -1.],
2022-05-18T03:52:02.4603330Z         [-1.,  1.],
2022-05-18T03:52:02.4604201Z         [-1.,  1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([-1., -1., -1., -1.])). Grads dict has 2 entries: {Parameter containing:
2022-05-18T03:52:02.4605148Z tensor([[-1.,  1.,  1.]], requires_grad=True): tensor([[  0., -28., -28.]]), Parameter containing:
2022-05-18T03:52:02.4605731Z tensor([[-1.,  1.],
2022-05-18T03:52:02.4606286Z         [ 1.,  1.]], requires_grad=True): tensor([[ 16., -16.],
2022-05-18T03:52:02.4606773Z         [ 16., -16.]])}
2022-05-18T03:52:02.4607720Z 2022-05-18 03:52:02,459 ddp_under_dist_autograd_test.py:261 INFO p:process 3 t:Dummy-3: Loss is 0.0 for mini batch: FeatureSet(dense_features=tensor([[ 1.,  1.],
2022-05-18T03:52:02.4608325Z         [ 1.,  1.],
2022-05-18T03:52:02.4608737Z         [ 1., -1.],
2022-05-18T03:52:02.4609579Z         [ 1., -1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([1., 1., 1., 1.])). Grads dict has 2 entries: {Parameter containing:
2022-05-18T03:52:02.4610479Z tensor([[-1.,  1.,  1.]], requires_grad=True): tensor([[56., 28., 28.]]), Parameter containing:
2022-05-18T03:52:02.4611021Z tensor([[-1.,  1.],
2022-05-18T03:52:02.4611478Z         [ 1.,  1.]], requires_grad=True): tensor([[32., 16.],
2022-05-18T03:52:02.4611899Z         [ 0., 16.]])}
2022-05-18T03:52:02.4621319Z 2022-05-18 03:52:02,461 ddp_under_dist_autograd_test.py:159 INFO p:process 2 t:Dummy-2: HybridModel has 2 groups of parameters.
2022-05-18T03:52:02.4622599Z 2022-05-18 03:52:02,461 ddp_under_dist_autograd_test.py:211 INFO p:process 2 t:Dummy-2: Succeeded in creating a HybridModel instance with 0 ddp params and 2 other local params.
2022-05-18T03:52:02.4814209Z 2022-05-18 03:52:02,480 ddp_under_dist_autograd_test.py:261 INFO p:process 2 t:Dummy-3: Loss is 56.0 for mini batch: FeatureSet(dense_features=tensor([[-1.,  1.],
2022-05-18T03:52:02.4814763Z         [-1.,  1.],
2022-05-18T03:52:02.4815166Z         [-1., -1.],
2022-05-18T03:52:02.4816011Z         [-1., -1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([1., 1., 1., 1.])). Grads dict has 2 entries: {Parameter containing:
2022-05-18T03:52:02.4816638Z tensor([[-1.,  1.,  1.]], requires_grad=True): tensor([[ 0., 28., 28.]]), Parameter containing:
2022-05-18T03:52:02.4817489Z tensor([[-1.,  1.],
2022-05-18T03:52:02.4818047Z         [ 1.,  1.]], requires_grad=True): tensor([[-16.,  16.],
2022-05-18T03:52:02.4818706Z         [-16.,  16.]])}
2022-05-18T03:52:02.4978117Z 2022-05-18 03:52:02,497 ddp_under_dist_autograd_test.py:261 INFO p:process 1 t:Dummy-4: Loss is -56.0 for mini batch: FeatureSet(dense_features=tensor([[-1., -1.],
2022-05-18T03:52:02.4980098Z         [-1., -1.],
2022-05-18T03:52:02.4980575Z         [-1.,  1.],
2022-05-18T03:52:02.4981576Z         [-1.,  1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([-1., -1., -1., -1.])). Grads dict has 2 entries: {Parameter containing:
2022-05-18T03:52:02.4982624Z tensor([[-1.,  1.,  1.]], requires_grad=True): tensor([[  0., -28., -28.]]), Parameter containing:
2022-05-18T03:52:02.4983459Z tensor([[-1.,  1.],
2022-05-18T03:52:02.4984119Z         [ 1.,  1.]], requires_grad=True): tensor([[ 16., -16.],
2022-05-18T03:52:02.4984703Z         [ 16., -16.]])}
2022-05-18T03:52:02.4985651Z 2022-05-18 03:52:02,497 ddp_under_dist_autograd_test.py:261 INFO p:process 0 t:Dummy-4: Loss is 0.0 for mini batch: FeatureSet(dense_features=tensor([[ 1., -1.],
2022-05-18T03:52:02.4986439Z         [ 1., -1.],
2022-05-18T03:52:02.4986822Z         [ 1.,  1.],
2022-05-18T03:52:02.4987770Z         [ 1.,  1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([-1., -1., -1., -1.])). Grads dict has 2 entries: {Parameter containing:
2022-05-18T03:52:02.4988584Z tensor([[-1.,  1.,  1.]], requires_grad=True): tensor([[-56., -28., -28.]]), Parameter containing:
2022-05-18T03:52:02.4989064Z tensor([[-1.,  1.],
2022-05-18T03:52:02.4989532Z         [ 1.,  1.]], requires_grad=True): tensor([[-32., -16.],
2022-05-18T03:52:02.4989932Z         [  0., -16.]])}
2022-05-18T03:52:02.5019082Z 2022-05-18 03:52:02,501 ddp_under_dist_autograd_test.py:261 INFO p:process 3 t:Dummy-4: Loss is 0.0 for mini batch: FeatureSet(dense_features=tensor([[ 1.,  1.],
2022-05-18T03:52:02.5019748Z         [ 1.,  1.],
2022-05-18T03:52:02.5021915Z         [ 1., -1.],
2022-05-18T03:52:02.5024219Z         [ 1., -1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([1., 1., 1., 1.])). Grads dict has 2 entries: {Parameter containing:
2022-05-18T03:52:02.5025159Z tensor([[-1.,  1.,  1.]], requires_grad=True): tensor([[56., 28., 28.]]), Parameter containing:
2022-05-18T03:52:02.5025737Z tensor([[-1.,  1.],
2022-05-18T03:52:02.5026184Z         [ 1.,  1.]], requires_grad=True): tensor([[32., 16.],
2022-05-18T03:52:02.5026606Z         [ 0., 16.]])}
2022-05-18T03:52:02.5044480Z 2022-05-18 03:52:02,503 ddp_under_dist_autograd_test.py:261 INFO p:process 2 t:Dummy-4: Loss is 56.0 for mini batch: FeatureSet(dense_features=tensor([[-1.,  1.],
2022-05-18T03:52:02.5045240Z         [-1.,  1.],
2022-05-18T03:52:02.5045659Z         [-1., -1.],
2022-05-18T03:52:02.5046488Z         [-1., -1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([1., 1., 1., 1.])). Grads dict has 2 entries: {Parameter containing:
2022-05-18T03:52:02.5047396Z tensor([[-1.,  1.,  1.]], requires_grad=True): tensor([[ 0., 28., 28.]]), Parameter containing:
2022-05-18T03:52:02.5047941Z tensor([[-1.,  1.],
2022-05-18T03:52:02.5048525Z         [ 1.,  1.]], requires_grad=True): tensor([[-16.,  16.],
2022-05-18T03:52:02.5049019Z         [-16.,  16.]])}
2022-05-18T03:52:02.5184409Z 2022-05-18 03:52:02,517 ddp_under_dist_autograd_test.py:261 INFO p:process 0 t:Dummy-5: Loss is 0.0 for mini batch: FeatureSet(dense_features=tensor([[ 1., -1.],
2022-05-18T03:52:02.5185226Z         [ 1., -1.],
2022-05-18T03:52:02.5185579Z         [ 1.,  1.],
2022-05-18T03:52:02.5186450Z         [ 1.,  1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([-1., -1., -1., -1.])). Grads dict has 2 entries: {Parameter containing:
2022-05-18T03:52:02.5187355Z tensor([[-1.,  1.,  1.]], requires_grad=True): tensor([[-56., -28., -28.]]), Parameter containing:
2022-05-18T03:52:02.5187891Z tensor([[-1.,  1.],
2022-05-18T03:52:02.5188478Z         [ 1.,  1.]], requires_grad=True): tensor([[-32., -16.],
2022-05-18T03:52:02.5189312Z         [  0., -16.]])}
2022-05-18T03:52:02.5194267Z 2022-05-18 03:52:02,518 ddp_under_dist_autograd_test.py:261 INFO p:process 1 t:Dummy-5: Loss is -56.0 for mini batch: FeatureSet(dense_features=tensor([[-1., -1.],
2022-05-18T03:52:02.5195032Z         [-1., -1.],
2022-05-18T03:52:02.5195467Z         [-1.,  1.],
2022-05-18T03:52:02.5196298Z         [-1.,  1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([-1., -1., -1., -1.])). Grads dict has 2 entries: {Parameter containing:
2022-05-18T03:52:02.5197126Z tensor([[-1.,  1.,  1.]], requires_grad=True): tensor([[  0., -28., -28.]]), Parameter containing:
2022-05-18T03:52:02.5197541Z tensor([[-1.,  1.],
2022-05-18T03:52:02.5201047Z         [ 1.,  1.]], requires_grad=True): tensor([[ 16., -16.],
2022-05-18T03:52:02.5201567Z         [ 16., -16.]])}
2022-05-18T03:52:02.5202533Z 2022-05-18 03:52:02,519 ddp_under_dist_autograd_test.py:261 INFO p:process 3 t:Dummy-5: Loss is 0.0 for mini batch: FeatureSet(dense_features=tensor([[ 1.,  1.],
2022-05-18T03:52:02.5203173Z         [ 1.,  1.],
2022-05-18T03:52:02.5203568Z         [ 1., -1.],
2022-05-18T03:52:02.5204404Z         [ 1., -1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([1., 1., 1., 1.])). Grads dict has 2 entries: {Parameter containing:
2022-05-18T03:52:02.5205299Z tensor([[-1.,  1.,  1.]], requires_grad=True): tensor([[56., 28., 28.]]), Parameter containing:
2022-05-18T03:52:02.5205862Z tensor([[-1.,  1.],
2022-05-18T03:52:02.5206303Z         [ 1.,  1.]], requires_grad=True): tensor([[32., 16.],
2022-05-18T03:52:02.5206727Z         [ 0., 16.]])}
2022-05-18T03:52:02.5267695Z 2022-05-18 03:52:02,526 ddp_under_dist_autograd_test.py:261 INFO p:process 2 t:Dummy-5: Loss is 56.0 for mini batch: FeatureSet(dense_features=tensor([[-1.,  1.],
2022-05-18T03:52:02.5268380Z         [-1.,  1.],
2022-05-18T03:52:02.5268802Z         [-1., -1.],
2022-05-18T03:52:02.5271254Z         [-1., -1.]]), sparse_features=tensor([0, 1, 0, 1]), values=tensor([1., 1., 1., 1.])). Grads dict has 2 entries: {Parameter containing:
2022-05-18T03:52:02.5272455Z tensor([[-1.,  1.,  1.]], requires_grad=True): tensor([[ 0., 28., 28.]]), Parameter containing:
2022-05-18T03:52:02.5273407Z tensor([[-1.,  1.],
2022-05-18T03:52:02.5274191Z         [ 1.,  1.]], requires_grad=True): tensor([[-16.,  16.],
2022-05-18T03:52:02.5275022Z         [-16.,  16.]])}
2022-05-18T03:52:02.5329532Z 2022-05-18 03:52:02,532 ddp_under_dist_autograd_test.py:364 INFO p:process 0 t:MainThread: Exiting the trainer #0...
2022-05-18T03:52:02.5335994Z 2022-05-18 03:52:02,533 ddp_under_dist_autograd_test.py:364 INFO p:process 1 t:MainThread: Exiting the trainer #1...
2022-05-18T03:52:02.5343239Z 2022-05-18 03:52:02,533 ddp_under_dist_autograd_test.py:364 INFO p:process 2 t:MainThread: Exiting the trainer #2...
2022-05-18T03:52:02.5352230Z 2022-05-18 03:52:02,534 ddp_under_dist_autograd_test.py:364 INFO p:process 3 t:MainThread: Exiting the trainer #3...
2022-05-18T03:52:02.5358902Z 2022-05-18 03:52:02,535 ddp_under_dist_autograd_test.py:344 INFO p:process 4 t:MainThread: Exiting remote worker.
2022-05-18T03:52:02.9232957Z ok (2.632s)
2022-05-18T03:52:02.9233167Z 
2022-05-18T03:52:02.9233731Z ----------------------------------------------------------------------
2022-05-18T03:52:02.9234099Z Ran 1 test in 2.633s
2022-05-18T03:52:02.9234214Z 
2022-05-18T03:52:02.9234274Z OK
2022-05-18T03:52:02.9234365Z 
2022-05-18T03:52:02.9234457Z Generating XML reports...
2022-05-18T03:52:02.9267356Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDdpUnderDistAutogradTest-20220518035200.xml
2022-05-18T03:52:03.6903143Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxu0coiw8
2022-05-18T03:52:03.6903679Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxu0coiw8/_remote_module_non_scriptable.py
2022-05-18T03:52:03.9414684Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T03:52:03.9424566Z 
2022-05-18T03:52:03.9424697Z Running tests...
2022-05-18T03:52:03.9425310Z ----------------------------------------------------------------------
2022-05-18T03:52:03.9430938Z   test_async_dist_autograd (__main__.TensorPipeDistAutogradTest)
2022-05-18T03:52:04.2563757Z This test ensures async processing for distributed autograd works ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9186
2022-05-18T03:52:04.2586317Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9187
2022-05-18T03:52:04.2609502Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9188
2022-05-18T03:52:04.2633870Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9189
2022-05-18T03:52:04.8860944Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr68mcvx2
2022-05-18T03:52:04.8861725Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr68mcvx2/_remote_module_non_scriptable.py
2022-05-18T03:52:04.9392383Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4qkc0nez
2022-05-18T03:52:04.9393251Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4qkc0nez/_remote_module_non_scriptable.py
2022-05-18T03:52:04.9586572Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnpbuepr0
2022-05-18T03:52:04.9587549Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnpbuepr0/_remote_module_non_scriptable.py
2022-05-18T03:52:04.9636679Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpalzq_rqd
2022-05-18T03:52:04.9638322Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpalzq_rqd/_remote_module_non_scriptable.py
2022-05-18T03:52:05.1360022Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:52:05.1861274Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:52:05.2074917Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:52:05.2125013Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:52:05.4469738Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:52:05.4570063Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:52:05.4671490Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:52:05.4672466Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:52:05.4673447Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:52:05.4674347Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:52:05.4675132Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:52:05.4679585Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:52:05.8678302Z ok (1.925s)
2022-05-18T03:52:05.8678530Z 
2022-05-18T03:52:05.8678981Z ----------------------------------------------------------------------
2022-05-18T03:52:05.8679388Z Ran 1 test in 1.925s
2022-05-18T03:52:05.8679563Z 
2022-05-18T03:52:05.8679658Z OK
2022-05-18T03:52:05.8679806Z 
2022-05-18T03:52:05.8679949Z Generating XML reports...
2022-05-18T03:52:05.8715078Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220518035203.xml
2022-05-18T03:52:06.6375834Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4w8lpj49
2022-05-18T03:52:06.6376384Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4w8lpj49/_remote_module_non_scriptable.py
2022-05-18T03:52:06.8891148Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T03:52:06.8900856Z 
2022-05-18T03:52:06.8901000Z Running tests...
2022-05-18T03:52:06.8901392Z ----------------------------------------------------------------------
2022-05-18T03:52:07.2016572Z   test_autograd_context (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9497
2022-05-18T03:52:07.2040404Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9498
2022-05-18T03:52:07.2063369Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9499
2022-05-18T03:52:07.2088441Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9500
2022-05-18T03:52:07.7806033Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpegbaz86b
2022-05-18T03:52:07.7806836Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpegbaz86b/_remote_module_non_scriptable.py
2022-05-18T03:52:07.7813986Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0dn6os31
2022-05-18T03:52:07.7815334Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0dn6os31/_remote_module_non_scriptable.py
2022-05-18T03:52:07.7869618Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkv000xg8
2022-05-18T03:52:07.7871597Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkv000xg8/_remote_module_non_scriptable.py
2022-05-18T03:52:07.8437841Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptu5xdf__
2022-05-18T03:52:07.8438612Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptu5xdf__/_remote_module_non_scriptable.py
2022-05-18T03:52:08.0279641Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:52:08.0280428Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:52:08.0355477Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:52:08.0892701Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:52:09.0134760Z ok (2.123s)
2022-05-18T03:52:09.0135062Z 
2022-05-18T03:52:09.0135432Z ----------------------------------------------------------------------
2022-05-18T03:52:09.0135683Z Ran 1 test in 2.123s
2022-05-18T03:52:09.0135813Z 
2022-05-18T03:52:09.0135875Z OK
2022-05-18T03:52:09.0135968Z 
2022-05-18T03:52:09.0136066Z Generating XML reports...
2022-05-18T03:52:09.0169078Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220518035206.xml
2022-05-18T03:52:09.7794122Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpac55phk2
2022-05-18T03:52:09.7794869Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpac55phk2/_remote_module_non_scriptable.py
2022-05-18T03:52:10.0316460Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T03:52:10.0325825Z 
2022-05-18T03:52:10.0326053Z Running tests...
2022-05-18T03:52:10.0326484Z ----------------------------------------------------------------------
2022-05-18T03:52:10.3537859Z   test_backward_accumulate_grads (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9716
2022-05-18T03:52:10.3559538Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9717
2022-05-18T03:52:10.3582584Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9718
2022-05-18T03:52:10.3607933Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9719
2022-05-18T03:52:11.0024072Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeaxmsc4v
2022-05-18T03:52:11.0025046Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeaxmsc4v/_remote_module_non_scriptable.py
2022-05-18T03:52:11.0257040Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptu3a9n_3
2022-05-18T03:52:11.0257838Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptu3a9n_3/_remote_module_non_scriptable.py
2022-05-18T03:52:11.0580105Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx1vtg8ul
2022-05-18T03:52:11.0580853Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx1vtg8ul/_remote_module_non_scriptable.py
2022-05-18T03:52:11.0840332Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphminna9c
2022-05-18T03:52:11.0841047Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphminna9c/_remote_module_non_scriptable.py
2022-05-18T03:52:11.2525962Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:52:11.2777139Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:52:11.3060709Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:52:11.3299253Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:52:11.7647386Z ok (1.732s)
2022-05-18T03:52:11.7647626Z 
2022-05-18T03:52:11.7648165Z ----------------------------------------------------------------------
2022-05-18T03:52:11.7648513Z Ran 1 test in 1.732s
2022-05-18T03:52:11.7648628Z 
2022-05-18T03:52:11.7648690Z OK
2022-05-18T03:52:11.7648781Z 
2022-05-18T03:52:11.7648873Z Generating XML reports...
2022-05-18T03:52:11.7682601Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220518035210.xml
2022-05-18T03:52:12.5363545Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp57xfx87z
2022-05-18T03:52:12.5364138Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp57xfx87z/_remote_module_non_scriptable.py
2022-05-18T03:52:12.7870193Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T03:52:12.7878986Z 
2022-05-18T03:52:12.7879179Z Running tests...
2022-05-18T03:52:12.7879526Z ----------------------------------------------------------------------
2022-05-18T03:52:13.1013754Z   test_backward_autograd_engine_error (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9955
2022-05-18T03:52:13.1036588Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9956
2022-05-18T03:52:13.1059734Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9957
2022-05-18T03:52:13.1083688Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9958
2022-05-18T03:52:13.7756755Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdpbddstl
2022-05-18T03:52:13.7758138Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdpbddstl/_remote_module_non_scriptable.py
2022-05-18T03:52:13.8021277Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxjvwosgo
2022-05-18T03:52:13.8022038Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxjvwosgo/_remote_module_non_scriptable.py
2022-05-18T03:52:13.8395976Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpru8twq9x
2022-05-18T03:52:13.8397047Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpru8twq9x/_remote_module_non_scriptable.py
2022-05-18T03:52:13.8544315Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5cv74gdy
2022-05-18T03:52:13.8545057Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5cv74gdy/_remote_module_non_scriptable.py
2022-05-18T03:52:14.0245366Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:52:14.0499708Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:52:14.0865741Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:52:14.1020980Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:52:14.7126147Z ok (1.924s)
2022-05-18T03:52:14.7126453Z 
2022-05-18T03:52:14.7126970Z ----------------------------------------------------------------------
2022-05-18T03:52:14.7127229Z Ran 1 test in 1.925s
2022-05-18T03:52:14.7127346Z 
2022-05-18T03:52:14.7127412Z OK
2022-05-18T03:52:14.7127491Z 
2022-05-18T03:52:14.7127589Z Generating XML reports...
2022-05-18T03:52:14.7163419Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220518035212.xml
2022-05-18T03:52:15.4806846Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprw1q4oxu
2022-05-18T03:52:15.4807648Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprw1q4oxu/_remote_module_non_scriptable.py
2022-05-18T03:52:15.7335971Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T03:52:15.7345720Z 
2022-05-18T03:52:15.7345851Z Running tests...
2022-05-18T03:52:15.7346428Z ----------------------------------------------------------------------
2022-05-18T03:52:16.0470232Z   test_backward_complex_python_udf (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10194
2022-05-18T03:52:16.0492328Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10195
2022-05-18T03:52:16.0515534Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10196
2022-05-18T03:52:16.0540278Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10197
2022-05-18T03:52:16.6519381Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpoejh74m6
2022-05-18T03:52:16.6520205Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpoejh74m6/_remote_module_non_scriptable.py
2022-05-18T03:52:16.6952310Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3ba7isq2
2022-05-18T03:52:16.6953082Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3ba7isq2/_remote_module_non_scriptable.py
2022-05-18T03:52:16.6999687Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpicuggoyg
2022-05-18T03:52:16.7001035Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpicuggoyg/_remote_module_non_scriptable.py
2022-05-18T03:52:16.7042540Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_uezr4_v
2022-05-18T03:52:16.7045017Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_uezr4_v/_remote_module_non_scriptable.py
2022-05-18T03:52:16.8992636Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:52:16.9443023Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:52:16.9475501Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:52:16.9530588Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:52:17.3578613Z ok (1.623s)
2022-05-18T03:52:17.3578847Z 
2022-05-18T03:52:17.3579390Z ----------------------------------------------------------------------
2022-05-18T03:52:17.3579687Z Ran 1 test in 1.623s
2022-05-18T03:52:17.3579803Z 
2022-05-18T03:52:17.3579866Z OK
2022-05-18T03:52:17.3579959Z 
2022-05-18T03:52:17.3580055Z Generating XML reports...
2022-05-18T03:52:17.3613638Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220518035215.xml
2022-05-18T03:52:18.1341525Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgql3rrs5
2022-05-18T03:52:18.1342527Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgql3rrs5/_remote_module_non_scriptable.py
2022-05-18T03:52:18.3853043Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T03:52:18.3862743Z 
2022-05-18T03:52:18.3863269Z Running tests...
2022-05-18T03:52:18.3863919Z ----------------------------------------------------------------------
2022-05-18T03:52:18.7005607Z   test_backward_different_dtypes (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10433
2022-05-18T03:52:18.7028635Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10434
2022-05-18T03:52:18.7051139Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10435
2022-05-18T03:52:18.7074856Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10436
2022-05-18T03:52:19.3556710Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqs6m0ge2
2022-05-18T03:52:19.3557508Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqs6m0ge2/_remote_module_non_scriptable.py
2022-05-18T03:52:19.3620918Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjbnr82_7
2022-05-18T03:52:19.3622451Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjbnr82_7/_remote_module_non_scriptable.py
2022-05-18T03:52:19.4274999Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6jhr5_zt
2022-05-18T03:52:19.4276167Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6jhr5_zt/_remote_module_non_scriptable.py
2022-05-18T03:52:19.4280716Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdcb2g8ht
2022-05-18T03:52:19.4283085Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdcb2g8ht/_remote_module_non_scriptable.py
2022-05-18T03:52:19.6031848Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:52:19.6086476Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:52:19.6747339Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:52:19.6759889Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:52:20.1116524Z ok (1.725s)
2022-05-18T03:52:20.1116787Z 
2022-05-18T03:52:20.1117128Z ----------------------------------------------------------------------
2022-05-18T03:52:20.1117367Z Ran 1 test in 1.725s
2022-05-18T03:52:20.1117488Z 
2022-05-18T03:52:20.1117550Z OK
2022-05-18T03:52:20.1117644Z 
2022-05-18T03:52:20.1117744Z Generating XML reports...
2022-05-18T03:52:20.1150966Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220518035218.xml
2022-05-18T03:52:20.8841476Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp48_a594i
2022-05-18T03:52:20.8842232Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp48_a594i/_remote_module_non_scriptable.py
2022-05-18T03:52:21.1382647Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T03:52:21.1392746Z 
2022-05-18T03:52:21.1393065Z Running tests...
2022-05-18T03:52:21.1393701Z ----------------------------------------------------------------------
2022-05-18T03:52:21.4607809Z   test_backward_different_tensor_dims (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10672
2022-05-18T03:52:21.4630692Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10673
2022-05-18T03:52:21.4653783Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10674
2022-05-18T03:52:21.4677987Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10675
2022-05-18T03:52:22.0433358Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpogg0r2l3
2022-05-18T03:52:22.0434359Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpogg0r2l3/_remote_module_non_scriptable.py
2022-05-18T03:52:22.0770404Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbp03a0r0
2022-05-18T03:52:22.0771530Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbp03a0r0/_remote_module_non_scriptable.py
2022-05-18T03:52:22.1019729Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt2y0c95s
2022-05-18T03:52:22.1020525Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsymry_5p
2022-05-18T03:52:22.1021965Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt2y0c95s/_remote_module_non_scriptable.py
2022-05-18T03:52:22.1022735Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsymry_5p/_remote_module_non_scriptable.py
2022-05-18T03:52:22.2909827Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:52:22.3265073Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:52:22.3501631Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:52:22.3526364Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:52:22.9719134Z ok (1.832s)
2022-05-18T03:52:22.9719415Z 
2022-05-18T03:52:22.9719844Z ----------------------------------------------------------------------
2022-05-18T03:52:22.9720107Z Ran 1 test in 1.833s
2022-05-18T03:52:22.9720224Z 
2022-05-18T03:52:22.9720316Z OK
2022-05-18T03:52:22.9720427Z 
2022-05-18T03:52:22.9720524Z Generating XML reports...
2022-05-18T03:52:22.9754069Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220518035221.xml
2022-05-18T03:52:23.7391410Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplgfvmrqi
2022-05-18T03:52:23.7391924Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplgfvmrqi/_remote_module_non_scriptable.py
2022-05-18T03:52:23.9899914Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T03:52:23.9909813Z 
2022-05-18T03:52:23.9909947Z Running tests...
2022-05-18T03:52:23.9910476Z ----------------------------------------------------------------------
2022-05-18T03:52:24.3029708Z   test_backward_invalid_args (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10911
2022-05-18T03:52:24.3050947Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10912
2022-05-18T03:52:24.3073917Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10913
2022-05-18T03:52:24.3098015Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10914
2022-05-18T03:52:24.9215676Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2htgwutu
2022-05-18T03:52:24.9216832Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2htgwutu/_remote_module_non_scriptable.py
2022-05-18T03:52:24.9349431Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5is6ur2k
2022-05-18T03:52:24.9350624Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5is6ur2k/_remote_module_non_scriptable.py
2022-05-18T03:52:24.9777786Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmproji4itf
2022-05-18T03:52:24.9778606Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmproji4itf/_remote_module_non_scriptable.py
2022-05-18T03:52:25.0094106Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_mf4zcdb
2022-05-18T03:52:25.0094897Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_mf4zcdb/_remote_module_non_scriptable.py
2022-05-18T03:52:25.1707692Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:52:25.1829475Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:52:25.2248135Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:52:25.2567163Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:52:25.7136181Z ok (1.722s)
2022-05-18T03:52:25.7136599Z 
2022-05-18T03:52:25.7137447Z ----------------------------------------------------------------------
2022-05-18T03:52:25.7137775Z Ran 1 test in 1.723s
2022-05-18T03:52:25.7137894Z 
2022-05-18T03:52:25.7137955Z OK
2022-05-18T03:52:25.7138046Z 
2022-05-18T03:52:25.7138148Z Generating XML reports...
2022-05-18T03:52:25.7171460Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220518035223.xml
2022-05-18T03:52:26.4821881Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3t2ket9t
2022-05-18T03:52:26.4822662Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3t2ket9t/_remote_module_non_scriptable.py
2022-05-18T03:52:26.7330265Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T03:52:26.7340087Z 
2022-05-18T03:52:26.7340177Z Running tests...
2022-05-18T03:52:26.7340887Z ----------------------------------------------------------------------
2022-05-18T03:52:27.0406958Z   test_backward_multiple_output_tensors (__main__.TensorPipeDistAutogradTest) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77516 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure IN_CI is not set and you are not using the flag --import-disabled-tests. (0.306s)
2022-05-18T03:52:27.0407480Z 
2022-05-18T03:52:27.0407731Z ----------------------------------------------------------------------
2022-05-18T03:52:27.0407993Z Ran 1 test in 0.307s
2022-05-18T03:52:27.0408108Z 
2022-05-18T03:52:27.0408181Z OK (skipped=1)
2022-05-18T03:52:27.0408325Z 
2022-05-18T03:52:27.0408439Z Generating XML reports...
2022-05-18T03:52:27.0430771Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220518035226.xml
2022-05-18T03:52:27.7521510Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpertnrag9
2022-05-18T03:52:27.7522287Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpertnrag9/_remote_module_non_scriptable.py
2022-05-18T03:52:28.0038998Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T03:52:28.0048831Z 
2022-05-18T03:52:28.0048963Z Running tests...
2022-05-18T03:52:28.0049413Z ----------------------------------------------------------------------
2022-05-18T03:52:28.3188594Z   test_backward_multiple_roots (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11144
2022-05-18T03:52:28.3210160Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11145
2022-05-18T03:52:28.3233613Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11146
2022-05-18T03:52:28.3258223Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11147
2022-05-18T03:52:28.9515099Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt5vn0tt5
2022-05-18T03:52:28.9516070Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt5vn0tt5/_remote_module_non_scriptable.py
2022-05-18T03:52:28.9586910Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp65q0yk3o
2022-05-18T03:52:28.9588513Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp65q0yk3o/_remote_module_non_scriptable.py
2022-05-18T03:52:28.9609013Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo_r0j91p
2022-05-18T03:52:28.9611299Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo_r0j91p/_remote_module_non_scriptable.py
2022-05-18T03:52:28.9745855Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdtn0scf2
2022-05-18T03:52:28.9747032Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdtn0scf2/_remote_module_non_scriptable.py
2022-05-18T03:52:29.2043656Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:52:29.2081559Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:52:29.2123412Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:52:29.2232051Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:52:29.7297722Z ok (1.725s)
2022-05-18T03:52:29.7298004Z 
2022-05-18T03:52:29.7298459Z ----------------------------------------------------------------------
2022-05-18T03:52:29.7298730Z Ran 1 test in 1.725s
2022-05-18T03:52:29.7298857Z 
2022-05-18T03:52:29.7298919Z OK
2022-05-18T03:52:29.7298998Z 
2022-05-18T03:52:29.7299099Z Generating XML reports...
2022-05-18T03:52:29.7331742Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220518035227.xml
2022-05-18T03:52:30.5027174Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqxrijuk0
2022-05-18T03:52:30.5027908Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqxrijuk0/_remote_module_non_scriptable.py
2022-05-18T03:52:30.7575251Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T03:52:30.7585085Z 
2022-05-18T03:52:30.7585179Z Running tests...
2022-05-18T03:52:30.7585922Z ----------------------------------------------------------------------
2022-05-18T03:52:31.0693828Z   test_backward_multiple_round_trips (__main__.TensorPipeDistAutogradTest) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77453 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure IN_CI is not set and you are not using the flag --import-disabled-tests. (0.311s)
2022-05-18T03:52:31.0694420Z 
2022-05-18T03:52:31.0694621Z ----------------------------------------------------------------------
2022-05-18T03:52:31.0694879Z Ran 1 test in 0.311s
2022-05-18T03:52:31.0694994Z 
2022-05-18T03:52:31.0695065Z OK (skipped=1)
2022-05-18T03:52:31.0695162Z 
2022-05-18T03:52:31.0695246Z Generating XML reports...
2022-05-18T03:52:31.0718438Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220518035230.xml
2022-05-18T03:52:31.7950588Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2_1i7vza
2022-05-18T03:52:31.7951668Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2_1i7vza/_remote_module_non_scriptable.py
2022-05-18T03:52:32.0470474Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T03:52:32.0480389Z 
2022-05-18T03:52:32.0480503Z Running tests...
2022-05-18T03:52:32.0481105Z ----------------------------------------------------------------------
2022-05-18T03:52:32.3650308Z   test_backward_no_grad_on_tensor (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11393
2022-05-18T03:52:32.3672341Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11394
2022-05-18T03:52:32.3695260Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11395
2022-05-18T03:52:32.3719331Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11396
2022-05-18T03:52:32.9599106Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjetufkgl
2022-05-18T03:52:32.9600131Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjetufkgl/_remote_module_non_scriptable.py
2022-05-18T03:52:32.9702010Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpilc9t_t5
2022-05-18T03:52:32.9703180Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpilc9t_t5/_remote_module_non_scriptable.py
2022-05-18T03:52:33.0016130Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprx4346m2
2022-05-18T03:52:33.0017534Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprx4346m2/_remote_module_non_scriptable.py
2022-05-18T03:52:33.0030102Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbi50b37n
2022-05-18T03:52:33.0032075Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbi50b37n/_remote_module_non_scriptable.py
2022-05-18T03:52:33.2075347Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:52:33.2166255Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:52:33.2487767Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:52:33.2508248Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:52:33.7759364Z ok (1.728s)
2022-05-18T03:52:33.7759586Z 
2022-05-18T03:52:33.7760105Z ----------------------------------------------------------------------
2022-05-18T03:52:33.7760561Z Ran 1 test in 1.728s
2022-05-18T03:52:33.7760689Z 
2022-05-18T03:52:33.7760751Z OK
2022-05-18T03:52:33.7760849Z 
2022-05-18T03:52:33.7760949Z Generating XML reports...
2022-05-18T03:52:33.7794651Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220518035232.xml
2022-05-18T03:52:34.5515079Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3uw34zbm
2022-05-18T03:52:34.5515742Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3uw34zbm/_remote_module_non_scriptable.py
2022-05-18T03:52:34.8056975Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T03:52:34.8066785Z 
2022-05-18T03:52:34.8066863Z Running tests...
2022-05-18T03:52:34.8067316Z ----------------------------------------------------------------------
2022-05-18T03:52:35.1226147Z   test_backward_node_failure (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11632
2022-05-18T03:52:35.1249792Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11633
2022-05-18T03:52:35.1272587Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11634
2022-05-18T03:52:35.1296645Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11635
2022-05-18T03:52:35.7413772Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1vkswjji
2022-05-18T03:52:35.7414555Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1vkswjji/_remote_module_non_scriptable.py
2022-05-18T03:52:35.7423590Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn0c8576l
2022-05-18T03:52:35.7425974Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn0c8576l/_remote_module_non_scriptable.py
2022-05-18T03:52:35.7636087Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyidnad4y
2022-05-18T03:52:35.7637658Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyidnad4y/_remote_module_non_scriptable.py
2022-05-18T03:52:35.7648630Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpurxkonlh
2022-05-18T03:52:35.7650217Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpurxkonlh/_remote_module_non_scriptable.py
2022-05-18T03:52:35.9893001Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:52:35.9909446Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:52:36.0107190Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:52:36.0109970Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:52:36.2878835Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:52:36.2977968Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:52:36.2978910Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:52:36.2980209Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:52:36.2981077Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:52:36.2982218Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:52:36.2987448Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:52:36.2991215Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:52:36.3216071Z [W tensorpipe_agent.cpp:728] RPC agent for worker0 encountered error when reading incoming request from worker3: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2022-05-18T03:52:36.3217505Z [W tensorpipe_agent.cpp:728] RPC agent for worker3 encountered error when reading incoming request from worker2: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2022-05-18T03:52:36.3218761Z [W tensorpipe_agent.cpp:728] RPC agent for worker3 encountered error when reading incoming request from worker0: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2022-05-18T03:52:36.3219784Z [W tensorpipe_agent.cpp:681] RPC agent for worker0 encountered error when sending response to request #2 to worker3: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2022-05-18T03:52:36.3220783Z [W tensorpipe_agent.cpp:728] RPC agent for worker1 encountered error when reading incoming request from worker0: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2022-05-18T03:52:36.3221874Z [W tensorpipe_agent.cpp:728] RPC agent for worker2 encountered error when reading incoming request from worker1: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2022-05-18T03:52:36.3224815Z [W tensorpipe_agent.cpp:918] RPC agent for worker2 encountered error when sending outgoing request #2 to worker1: EOF: end of file (this error originated at tensorpipe/transport/uv/connection_impl.cc:132)
2022-05-18T03:52:36.3225800Z [W tensorpipe_agent.cpp:918] RPC agent for worker2 encountered error when sending outgoing request #3 to worker1: EOF: end of file (this error originated at tensorpipe/transport/uv/connection_impl.cc:132)
2022-05-18T03:52:36.3226941Z [W tensorpipe_agent.cpp:728] RPC agent for worker0 encountered error when reading incoming request from worker1: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2022-05-18T03:52:36.3227845Z [W tensorpipe_agent.cpp:942] RPC agent for worker0 encountered error when reading incoming response from worker3: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2022-05-18T03:52:36.3228569Z [E container.cpp:257] Could not release Dist Autograd Context on node 0: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2022-05-18T03:52:36.3233576Z [W tensorpipe_agent.cpp:918] RPC agent for worker0 encountered error when sending outgoing request #4 to worker1: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2022-05-18T03:52:36.3236643Z [W tensorpipe_agent.cpp:918] RPC agent for worker2 encountered error when sending outgoing request #4 to worker3: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2022-05-18T03:52:36.3240868Z [W tensorpipe_agent.cpp:918] RPC agent for worker0 encountered error when sending outgoing request #6 to worker3: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2022-05-18T03:52:36.3245260Z [W tensorpipe_agent.cpp:918] RPC agent for worker2 encountered error when sending outgoing request #5 to worker3: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2022-05-18T03:52:36.3247876Z [W tensorpipe_agent.cpp:918] RPC agent for worker0 encountered error when sending outgoing request #7 to worker1: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2022-05-18T03:52:36.3267660Z [W tensorpipe_agent.cpp:918] RPC agent for worker2 encountered error when sending outgoing request #6 to worker1: EOF: end of file (this error originated at tensorpipe/transport/uv/connection_impl.cc:132)
2022-05-18T03:52:36.3268892Z [W tensorpipe_agent.cpp:918] RPC agent for worker2 encountered error when sending outgoing request #7 to worker3: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2022-05-18T03:52:36.3270850Z [W tensorpipe_agent.cpp:918] RPC agent for worker0 encountered error when sending outgoing request #8 to worker3: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2022-05-18T03:52:36.3272591Z [W tensorpipe_agent.cpp:918] RPC agent for worker0 encountered error when sending outgoing request #9 to worker1: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2022-05-18T03:52:36.3273685Z [E container.cpp:257] Could not release Dist Autograd Context on node 1: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2022-05-18T03:52:36.3274646Z [W tensorpipe_agent.cpp:728] RPC agent for worker2 encountered error when reading incoming request from worker0: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2022-05-18T03:52:36.3281154Z [W tensorpipe_agent.cpp:728] RPC agent for worker0 encountered error when reading incoming request from worker2: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2022-05-18T03:52:36.5336983Z ok (1.727s)
2022-05-18T03:52:36.5337228Z 
2022-05-18T03:52:36.5337705Z ----------------------------------------------------------------------
2022-05-18T03:52:36.5338078Z Ran 1 test in 1.727s
2022-05-18T03:52:36.5338323Z 
2022-05-18T03:52:36.5338430Z OK
2022-05-18T03:52:36.5338585Z 
2022-05-18T03:52:36.5338744Z Generating XML reports...
2022-05-18T03:52:36.5372364Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220518035234.xml
2022-05-18T03:52:37.3020739Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuzem5u_c
2022-05-18T03:52:37.3021482Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuzem5u_c/_remote_module_non_scriptable.py
2022-05-18T03:52:37.5576386Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T03:52:37.5586356Z 
2022-05-18T03:52:37.5586501Z Running tests...
2022-05-18T03:52:37.5587058Z ----------------------------------------------------------------------
2022-05-18T03:52:37.8743026Z   test_backward_node_failure_python_udf (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11873
2022-05-18T03:52:37.8765422Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11874
2022-05-18T03:52:37.8789258Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11875
2022-05-18T03:52:37.8813316Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11876
2022-05-18T03:52:38.5340073Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyift1j31
2022-05-18T03:52:38.5341075Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyift1j31/_remote_module_non_scriptable.py
2022-05-18T03:52:38.5379519Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe41bhfdb
2022-05-18T03:52:38.5382071Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe41bhfdb/_remote_module_non_scriptable.py
2022-05-18T03:52:38.7085969Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe814ep2w
2022-05-18T03:52:38.7086944Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe814ep2w/_remote_module_non_scriptable.py
2022-05-18T03:52:38.7137863Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpda69m87b
2022-05-18T03:52:38.7139470Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpda69m87b/_remote_module_non_scriptable.py
2022-05-18T03:52:38.7838816Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:52:38.7843092Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:52:38.9919852Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:52:38.9959887Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:52:39.2428575Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:52:39.2630782Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:52:39.2631792Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:52:39.2632327Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:52:39.2633329Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:52:39.2634092Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:52:39.2634971Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:52:39.2637949Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:52:39.2837898Z [W tensorpipe_agent.cpp:918] RPC agent for worker3 encountered error when sending outgoing request #4 to worker2: ECONNREFUSED: connection refused (this error originated at tensorpipe/transport/uv/connection_impl.cc:62)
2022-05-18T03:52:39.2839314Z [W tensorpipe_agent.cpp:728] RPC agent for worker2 encountered error when reading incoming request from worker1: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2022-05-18T03:52:39.2840458Z [W tensorpipe_agent.cpp:728] RPC agent for worker2 encountered error when reading incoming request from worker0: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2022-05-18T03:52:39.2841698Z [W tensorpipe_agent.cpp:728] RPC agent for worker3 encountered error when reading incoming request from worker2: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2022-05-18T03:52:39.2842722Z [W tensorpipe_agent.cpp:728] RPC agent for worker0 encountered error when reading incoming request from worker2: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2022-05-18T03:52:39.2843748Z [W tensorpipe_agent.cpp:942] RPC agent for worker0 encountered error when reading incoming response from worker2: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2022-05-18T03:52:39.2850841Z [E container.cpp:257] Could not release Dist Autograd Context on node 3: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2022-05-18T03:52:39.2868648Z [W tensorpipe_agent.cpp:918] RPC agent for worker1 encountered error when sending outgoing request #5 to worker2: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2022-05-18T03:52:39.2887437Z [W tensorpipe_agent.cpp:918] RPC agent for worker0 encountered error when sending outgoing request #10 to worker2: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2022-05-18T03:52:39.2889188Z [W tensorpipe_agent.cpp:918] RPC agent for worker1 encountered error when sending outgoing request #7 to worker2: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2022-05-18T03:52:39.2890968Z [W tensorpipe_agent.cpp:728] RPC agent for worker3 encountered error when reading incoming request from worker0: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2022-05-18T03:52:39.2892431Z [W tensorpipe_agent.cpp:728] RPC agent for worker1 encountered error when reading incoming request from worker0: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2022-05-18T03:52:39.2892978Z [W tensorpipe_agent.cpp:942] RPC agent for worker1 encountered error when reading incoming response from worker0: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2022-05-18T03:52:39.2897567Z [W tensorpipe_agent.cpp:728] RPC agent for worker0 encountered error when reading incoming request from worker1: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2022-05-18T03:52:39.2899118Z [W tensorpipe_agent.cpp:728] RPC agent for worker0 encountered error when reading incoming request from worker3: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2022-05-18T03:52:39.2900456Z [E container.cpp:257] Could not release Dist Autograd Context on node 1: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2022-05-18T03:52:39.2935775Z [W tensorpipe_agent.cpp:918] RPC agent for worker3 encountered error when sending outgoing request #6 to worker0: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2022-05-18T03:52:39.2936846Z [W tensorpipe_agent.cpp:918] RPC agent for worker1 encountered error when sending outgoing request #8 to worker2: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2022-05-18T03:52:39.2938307Z [E container.cpp:257] Could not release Dist Autograd Context on node 0: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2022-05-18T03:52:39.4858548Z ok (1.927s)
2022-05-18T03:52:39.4858780Z 
2022-05-18T03:52:39.4859297Z ----------------------------------------------------------------------
2022-05-18T03:52:39.4859728Z Ran 1 test in 1.927s
2022-05-18T03:52:39.4859870Z 
2022-05-18T03:52:39.4859939Z OK
2022-05-18T03:52:39.4860029Z 
2022-05-18T03:52:39.4860123Z Generating XML reports...
2022-05-18T03:52:39.4893695Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220518035237.xml
2022-05-18T03:52:40.2605626Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4ng61ucn
2022-05-18T03:52:40.2606172Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4ng61ucn/_remote_module_non_scriptable.py
2022-05-18T03:52:40.5144003Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T03:52:40.5153449Z 
2022-05-18T03:52:40.5153756Z Running tests...
2022-05-18T03:52:40.5154383Z ----------------------------------------------------------------------
2022-05-18T03:52:40.8284976Z   test_backward_python_udf_error (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12114
2022-05-18T03:52:40.8307613Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12115
2022-05-18T03:52:40.8330380Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12116
2022-05-18T03:52:40.8354685Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 12117
2022-05-18T03:52:41.4295425Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvexkmu9p
2022-05-18T03:52:41.4296344Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvexkmu9p/_remote_module_non_scriptable.py
2022-05-18T03:52:41.4648675Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqcrp64_k
2022-05-18T03:52:41.4649878Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqcrp64_k/_remote_module_non_scriptable.py
2022-05-18T03:52:41.4910646Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt9i39fsb
2022-05-18T03:52:41.4911390Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_8iyfay_
2022-05-18T03:52:41.4912093Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt9i39fsb/_remote_module_non_scriptable.py
2022-05-18T03:52:41.4912818Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_8iyfay_/_remote_module_non_scriptable.py
2022-05-18T03:52:41.6786832Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:52:41.7149303Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:52:41.7401353Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:52:41.7422122Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:52:42.2395319Z ok (1.724s)
2022-05-18T03:52:42.2395590Z 
2022-05-18T03:52:42.2396099Z ----------------------------------------------------------------------
2022-05-18T03:52:42.2396342Z Ran 1 test in 1.724s
2022-05-18T03:52:42.2396461Z 
2022-05-18T03:52:42.2396521Z OK
2022-05-18T03:52:42.2396612Z 
2022-05-18T03:52:42.2396705Z Generating XML reports...
2022-05-18T03:52:42.2429945Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220518035240.xml
2022-05-18T03:52:43.0218992Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplokxqvm1
2022-05-18T03:52:43.0219627Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplokxqvm1/_remote_module_non_scriptable.py
2022-05-18T03:52:43.2755385Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T03:52:43.2764099Z 
2022-05-18T03:52:43.2764232Z Running tests...
2022-05-18T03:52:43.2764826Z ----------------------------------------------------------------------
2022-05-18T03:52:43.5935111Z   test_backward_rref (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12353
2022-05-18T03:52:43.5957437Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12354
2022-05-18T03:52:43.5980565Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12355
2022-05-18T03:52:43.6005757Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 12356
2022-05-18T03:52:44.1749227Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvnfbopwk
2022-05-18T03:52:44.1750015Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvnfbopwk/_remote_module_non_scriptable.py
2022-05-18T03:52:44.2261176Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvvhg2c5t
2022-05-18T03:52:44.2262008Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe2u65238
2022-05-18T03:52:44.2262724Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvvhg2c5t/_remote_module_non_scriptable.py
2022-05-18T03:52:44.2263877Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe2u65238/_remote_module_non_scriptable.py
2022-05-18T03:52:44.2389866Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqhp9h3if
2022-05-18T03:52:44.2391365Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqhp9h3if/_remote_module_non_scriptable.py
2022-05-18T03:52:44.4213023Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:52:44.4737756Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:52:44.4757617Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:52:44.4889042Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:52:45.0044398Z ok (1.728s)
2022-05-18T03:52:45.0044649Z 
2022-05-18T03:52:45.0045158Z ----------------------------------------------------------------------
2022-05-18T03:52:45.0045750Z Ran 1 test in 1.728s
2022-05-18T03:52:45.0045951Z 
2022-05-18T03:52:45.0046061Z OK
2022-05-18T03:52:45.0046229Z 
2022-05-18T03:52:45.0046398Z Generating XML reports...
2022-05-18T03:52:45.0080043Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220518035243.xml
2022-05-18T03:52:45.7722263Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7wfxkecs
2022-05-18T03:52:45.7722899Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7wfxkecs/_remote_module_non_scriptable.py
2022-05-18T03:52:46.0243568Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T03:52:46.0253414Z 
2022-05-18T03:52:46.0253526Z Running tests...
2022-05-18T03:52:46.0253976Z ----------------------------------------------------------------------
2022-05-18T03:52:46.3373603Z   test_backward_rref_multi (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12592
2022-05-18T03:52:46.3396410Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12593
2022-05-18T03:52:46.3418999Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12594
2022-05-18T03:52:46.3443373Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 12595
2022-05-18T03:52:46.9955258Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7z02p840
2022-05-18T03:52:46.9956070Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7z02p840/_remote_module_non_scriptable.py
2022-05-18T03:52:47.0065556Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzn9dozvi
2022-05-18T03:52:47.0066844Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzn9dozvi/_remote_module_non_scriptable.py
2022-05-18T03:52:47.0071384Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqbbsou3t
2022-05-18T03:52:47.0073337Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqbbsou3t/_remote_module_non_scriptable.py
2022-05-18T03:52:47.0206099Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp84gec5k
2022-05-18T03:52:47.0207717Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp84gec5k/_remote_module_non_scriptable.py
2022-05-18T03:52:47.2439587Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:52:47.2524111Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:52:47.2545118Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:52:47.2682067Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:52:47.7483894Z ok (1.723s)
2022-05-18T03:52:47.7484143Z 
2022-05-18T03:52:47.7484655Z ----------------------------------------------------------------------
2022-05-18T03:52:47.7485028Z Ran 1 test in 1.723s
2022-05-18T03:52:47.7485145Z 
2022-05-18T03:52:47.7485207Z OK
2022-05-18T03:52:47.7485298Z 
2022-05-18T03:52:47.7485395Z Generating XML reports...
2022-05-18T03:52:47.7518919Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220518035246.xml
2022-05-18T03:52:48.5143098Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi1yeom7r
2022-05-18T03:52:48.5143905Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi1yeom7r/_remote_module_non_scriptable.py
2022-05-18T03:52:48.7705447Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T03:52:48.7714907Z 
2022-05-18T03:52:48.7715146Z Running tests...
2022-05-18T03:52:48.7715554Z ----------------------------------------------------------------------
2022-05-18T03:52:49.0883818Z   test_backward_rref_nested (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12831
2022-05-18T03:52:49.0905937Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12832
2022-05-18T03:52:49.0929211Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12833
2022-05-18T03:52:49.0953297Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 12834
2022-05-18T03:52:49.6725196Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj7wqeh6t
2022-05-18T03:52:49.6725985Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj7wqeh6t/_remote_module_non_scriptable.py
2022-05-18T03:52:49.7329197Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2tlki0zj
2022-05-18T03:52:49.7329985Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk2uukjk1
2022-05-18T03:52:49.7330707Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2tlki0zj/_remote_module_non_scriptable.py
2022-05-18T03:52:49.7331414Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk2uukjk1/_remote_module_non_scriptable.py
2022-05-18T03:52:49.7381470Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp8auqxkl
2022-05-18T03:52:49.7382732Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp8auqxkl/_remote_module_non_scriptable.py
2022-05-18T03:52:49.9200158Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:52:49.9789148Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:52:49.9802149Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:52:49.9852561Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:52:50.4992817Z ok (1.727s)
2022-05-18T03:52:50.4993050Z 
2022-05-18T03:52:50.4993573Z ----------------------------------------------------------------------
2022-05-18T03:52:50.4993964Z Ran 1 test in 1.728s
2022-05-18T03:52:50.4994082Z 
2022-05-18T03:52:50.4994145Z OK
2022-05-18T03:52:50.4994241Z 
2022-05-18T03:52:50.4994927Z Generating XML reports...
2022-05-18T03:52:50.5030789Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220518035248.xml
2022-05-18T03:52:51.2674691Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgmfheh8r
2022-05-18T03:52:51.2675599Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgmfheh8r/_remote_module_non_scriptable.py
2022-05-18T03:52:51.5194759Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T03:52:51.5203806Z 
2022-05-18T03:52:51.5203941Z Running tests...
2022-05-18T03:52:51.5204517Z ----------------------------------------------------------------------
2022-05-18T03:52:51.8376533Z   test_backward_simple (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13070
2022-05-18T03:52:51.8399854Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13071
2022-05-18T03:52:51.8422688Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13072
2022-05-18T03:52:51.8448482Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 13073
2022-05-18T03:52:52.4550852Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprm7o5hlf
2022-05-18T03:52:52.4552418Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprm7o5hlf/_remote_module_non_scriptable.py
2022-05-18T03:52:52.4710799Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwifpwjrg
2022-05-18T03:52:52.4711830Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwifpwjrg/_remote_module_non_scriptable.py
2022-05-18T03:52:52.4801241Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5y_d_ne0
2022-05-18T03:52:52.4801995Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpubv735cy
2022-05-18T03:52:52.4803234Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5y_d_ne0/_remote_module_non_scriptable.py
2022-05-18T03:52:52.4803950Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpubv735cy/_remote_module_non_scriptable.py
2022-05-18T03:52:52.7050348Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:52:52.7173881Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:52:52.7268208Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:52:52.7307069Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:52:53.2486530Z ok (1.728s)
2022-05-18T03:52:53.2486700Z 
2022-05-18T03:52:53.2487104Z ----------------------------------------------------------------------
2022-05-18T03:52:53.2487442Z Ran 1 test in 1.728s
2022-05-18T03:52:53.2487562Z 
2022-05-18T03:52:53.2487630Z OK
2022-05-18T03:52:53.2487710Z 
2022-05-18T03:52:53.2487807Z Generating XML reports...
2022-05-18T03:52:53.2520945Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220518035251.xml
2022-05-18T03:52:54.0153337Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvmacsxa2
2022-05-18T03:52:54.0153884Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvmacsxa2/_remote_module_non_scriptable.py
2022-05-18T03:52:54.2672650Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T03:52:54.2755057Z 
2022-05-18T03:52:54.2755482Z Running tests...
2022-05-18T03:52:54.2755954Z ----------------------------------------------------------------------
2022-05-18T03:52:54.5835075Z   test_backward_simple_python_udf (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13309
2022-05-18T03:52:54.5856967Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13310
2022-05-18T03:52:54.5880566Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13311
2022-05-18T03:52:54.5904682Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 13312
2022-05-18T03:52:55.2193699Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxr0n4r47
2022-05-18T03:52:55.2194716Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxr0n4r47/_remote_module_non_scriptable.py
2022-05-18T03:52:55.2526683Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpekmdtaye
2022-05-18T03:52:55.2527657Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpekmdtaye/_remote_module_non_scriptable.py
2022-05-18T03:52:55.2572675Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwifl2hqj
2022-05-18T03:52:55.2574048Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwifl2hqj/_remote_module_non_scriptable.py
2022-05-18T03:52:55.3055609Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpds_oyo5f
2022-05-18T03:52:55.3056392Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpds_oyo5f/_remote_module_non_scriptable.py
2022-05-18T03:52:55.4707405Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:52:55.5005230Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:52:55.5064700Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:52:55.5510872Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:52:55.9945175Z ok (1.726s)
2022-05-18T03:52:55.9945366Z 
2022-05-18T03:52:55.9945819Z ----------------------------------------------------------------------
2022-05-18T03:52:55.9946282Z Ran 1 test in 1.726s
2022-05-18T03:52:55.9946497Z 
2022-05-18T03:52:55.9946591Z OK
2022-05-18T03:52:55.9946686Z 
2022-05-18T03:52:55.9946764Z Generating XML reports...
2022-05-18T03:52:55.9980039Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220518035254.xml
2022-05-18T03:52:56.7662024Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8ez1_ff7
2022-05-18T03:52:56.7663018Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8ez1_ff7/_remote_module_non_scriptable.py
2022-05-18T03:52:57.0203482Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T03:52:57.0212779Z 
2022-05-18T03:52:57.0212875Z Running tests...
2022-05-18T03:52:57.0213444Z ----------------------------------------------------------------------
2022-05-18T03:52:57.3432517Z   test_backward_simple_script_call (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13548
2022-05-18T03:52:57.3454993Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13549
2022-05-18T03:52:57.3478706Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13550
2022-05-18T03:52:57.3503898Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 13551
2022-05-18T03:52:57.9760986Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmak27t9f
2022-05-18T03:52:57.9762269Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmak27t9f/_remote_module_non_scriptable.py
2022-05-18T03:52:57.9809186Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpivizqtw5
2022-05-18T03:52:57.9811306Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpivizqtw5/_remote_module_non_scriptable.py
2022-05-18T03:52:57.9825525Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgfbw05u7
2022-05-18T03:52:57.9827440Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgfbw05u7/_remote_module_non_scriptable.py
2022-05-18T03:52:58.0403022Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5dizt0el
2022-05-18T03:52:58.0404028Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5dizt0el/_remote_module_non_scriptable.py
2022-05-18T03:52:58.2234568Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:52:58.2271207Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:52:58.2278968Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:52:58.2886701Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:52:58.9546381Z ok (1.933s)
2022-05-18T03:52:58.9546642Z 
2022-05-18T03:52:58.9547145Z ----------------------------------------------------------------------
2022-05-18T03:52:58.9547524Z Ran 1 test in 1.933s
2022-05-18T03:52:58.9547641Z 
2022-05-18T03:52:58.9547702Z OK
2022-05-18T03:52:58.9547795Z 
2022-05-18T03:52:58.9547893Z Generating XML reports...
2022-05-18T03:52:58.9580760Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220518035257.xml
2022-05-18T03:52:59.7132510Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprslkimrm
2022-05-18T03:52:59.7133608Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprslkimrm/_remote_module_non_scriptable.py
2022-05-18T03:52:59.9684362Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T03:52:59.9694577Z 
2022-05-18T03:52:59.9694706Z Running tests...
2022-05-18T03:52:59.9695282Z ----------------------------------------------------------------------
2022-05-18T03:53:00.2825233Z   test_backward_simple_self (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13787
2022-05-18T03:53:00.2848382Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13788
2022-05-18T03:53:00.2872162Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13789
2022-05-18T03:53:00.2896171Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 13790
2022-05-18T03:53:00.9419865Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplkfyxkuf
2022-05-18T03:53:00.9421116Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplkfyxkuf/_remote_module_non_scriptable.py
2022-05-18T03:53:00.9493548Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjaf8_esq
2022-05-18T03:53:00.9494943Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjaf8_esq/_remote_module_non_scriptable.py
2022-05-18T03:53:00.9657874Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpml1azfwo
2022-05-18T03:53:00.9659270Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpml1azfwo/_remote_module_non_scriptable.py
2022-05-18T03:53:00.9659911Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpne35wmy6
2022-05-18T03:53:00.9662627Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpne35wmy6/_remote_module_non_scriptable.py
2022-05-18T03:53:01.1905975Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:53:01.1966088Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:53:01.2131007Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:53:01.2133336Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:53:01.6935238Z ok (1.724s)
2022-05-18T03:53:01.6935514Z 
2022-05-18T03:53:01.6936068Z ----------------------------------------------------------------------
2022-05-18T03:53:01.6936399Z Ran 1 test in 1.724s
2022-05-18T03:53:01.6936517Z 
2022-05-18T03:53:01.6936579Z OK
2022-05-18T03:53:01.6936657Z 
2022-05-18T03:53:01.6936755Z Generating XML reports...
2022-05-18T03:53:01.6970388Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220518035259.xml
2022-05-18T03:53:02.4645860Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdzrrkk6r
2022-05-18T03:53:02.4646716Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdzrrkk6r/_remote_module_non_scriptable.py
2022-05-18T03:53:02.7188662Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T03:53:02.7197804Z 
2022-05-18T03:53:02.7197897Z Running tests...
2022-05-18T03:53:02.7198286Z ----------------------------------------------------------------------
2022-05-18T03:53:03.0315125Z   test_backward_unused_send_function (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14026
2022-05-18T03:53:03.0337336Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14027
2022-05-18T03:53:03.0361297Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14028
2022-05-18T03:53:03.0391993Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 14029
2022-05-18T03:53:03.6486340Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps9zig4du
2022-05-18T03:53:03.6487558Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps9zig4du/_remote_module_non_scriptable.py
2022-05-18T03:53:03.6593660Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbt3hnvjo
2022-05-18T03:53:03.6595127Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbt3hnvjo/_remote_module_non_scriptable.py
2022-05-18T03:53:03.7322405Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfdvwmd22
2022-05-18T03:53:03.7323154Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfdvwmd22/_remote_module_non_scriptable.py
2022-05-18T03:53:03.7357211Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo2rfz1vd
2022-05-18T03:53:03.7359176Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo2rfz1vd/_remote_module_non_scriptable.py
2022-05-18T03:53:03.8972350Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:53:03.9079314Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:53:03.9805340Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:53:03.9831572Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:53:14.4581706Z ok (11.738s)
2022-05-18T03:53:14.4581973Z 
2022-05-18T03:53:14.4582499Z ----------------------------------------------------------------------
2022-05-18T03:53:14.4583064Z Ran 1 test in 11.738s
2022-05-18T03:53:14.4583188Z 
2022-05-18T03:53:14.4583237Z OK
2022-05-18T03:53:14.4583331Z 
2022-05-18T03:53:14.4583426Z Generating XML reports...
2022-05-18T03:53:14.4617005Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220518035302.xml
2022-05-18T03:53:15.2294603Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9090x5_f
2022-05-18T03:53:15.2295331Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9090x5_f/_remote_module_non_scriptable.py
2022-05-18T03:53:15.4833595Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T03:53:15.4843429Z 
2022-05-18T03:53:15.4843663Z Running tests...
2022-05-18T03:53:15.4844337Z ----------------------------------------------------------------------
2022-05-18T03:53:15.7920696Z   test_backward_unused_tensors (__main__.TensorPipeDistAutogradTest) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77556 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure IN_CI is not set and you are not using the flag --import-disabled-tests. (0.307s)
2022-05-18T03:53:15.7921216Z 
2022-05-18T03:53:15.7921423Z ----------------------------------------------------------------------
2022-05-18T03:53:15.7921691Z Ran 1 test in 0.307s
2022-05-18T03:53:15.7921807Z 
2022-05-18T03:53:15.7921868Z OK (skipped=1)
2022-05-18T03:53:15.7921974Z 
2022-05-18T03:53:15.7922059Z Generating XML reports...
2022-05-18T03:53:15.7944116Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220518035315.xml
2022-05-18T03:53:16.5038708Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuioceqay
2022-05-18T03:53:16.5039161Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuioceqay/_remote_module_non_scriptable.py
2022-05-18T03:53:16.7549370Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T03:53:16.7558860Z 
2022-05-18T03:53:16.7559263Z Running tests...
2022-05-18T03:53:16.7559617Z ----------------------------------------------------------------------
2022-05-18T03:53:17.0701170Z   test_backward_verify_hooks (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14279
2022-05-18T03:53:17.0722909Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14280
2022-05-18T03:53:17.0746267Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14281
2022-05-18T03:53:17.0771379Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 14282
2022-05-18T03:53:17.7634397Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnea9fn05
2022-05-18T03:53:17.7635228Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnea9fn05/_remote_module_non_scriptable.py
2022-05-18T03:53:17.7960501Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzth4pezu
2022-05-18T03:53:17.7961271Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzth4pezu/_remote_module_non_scriptable.py
2022-05-18T03:53:17.7972843Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4tufhswc
2022-05-18T03:53:17.7974757Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4tufhswc/_remote_module_non_scriptable.py
2022-05-18T03:53:17.8206606Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3799_h64
2022-05-18T03:53:17.8208043Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3799_h64/_remote_module_non_scriptable.py
2022-05-18T03:53:18.0109118Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:53:18.0416989Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:53:18.0457456Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:53:18.0686388Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:53:18.5859720Z ok (1.830s)
2022-05-18T03:53:18.5859929Z 
2022-05-18T03:53:18.5860268Z ----------------------------------------------------------------------
2022-05-18T03:53:18.5860524Z Ran 1 test in 1.830s
2022-05-18T03:53:18.5860641Z 
2022-05-18T03:53:18.5860702Z OK
2022-05-18T03:53:18.5860796Z 
2022-05-18T03:53:18.5860890Z Generating XML reports...
2022-05-18T03:53:18.5893863Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220518035316.xml
2022-05-18T03:53:19.3499585Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphn_xh135
2022-05-18T03:53:19.3500127Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphn_xh135/_remote_module_non_scriptable.py
2022-05-18T03:53:19.6011231Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T03:53:19.6020294Z 
2022-05-18T03:53:19.6020405Z Running tests...
2022-05-18T03:53:19.6021188Z ----------------------------------------------------------------------
2022-05-18T03:53:19.9113079Z   test_backward_without_context (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14518
2022-05-18T03:53:19.9136363Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14519
2022-05-18T03:53:19.9159131Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14520
2022-05-18T03:53:19.9183070Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 14521
2022-05-18T03:53:20.6074431Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprlqkj592
2022-05-18T03:53:20.6075630Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprlqkj592/_remote_module_non_scriptable.py
2022-05-18T03:53:20.6169321Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8edhwomm
2022-05-18T03:53:20.6170462Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8edhwomm/_remote_module_non_scriptable.py
2022-05-18T03:53:20.6266900Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdmk3bo5s
2022-05-18T03:53:20.6267956Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdmk3bo5s/_remote_module_non_scriptable.py
2022-05-18T03:53:20.6268627Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8zv45zaf
2022-05-18T03:53:20.6271085Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8zv45zaf/_remote_module_non_scriptable.py
2022-05-18T03:53:20.8561237Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:53:20.8623263Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:53:20.8714728Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:53:20.8732816Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:53:21.3224187Z ok (1.720s)
2022-05-18T03:53:21.3224459Z 
2022-05-18T03:53:21.3224999Z ----------------------------------------------------------------------
2022-05-18T03:53:21.3225279Z Ran 1 test in 1.720s
2022-05-18T03:53:21.3225396Z 
2022-05-18T03:53:21.3225499Z OK
2022-05-18T03:53:21.3226373Z 
2022-05-18T03:53:21.3226631Z Generating XML reports...
2022-05-18T03:53:21.3260270Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220518035319.xml
2022-05-18T03:53:22.0897305Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcyafxtx5
2022-05-18T03:53:22.0898223Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcyafxtx5/_remote_module_non_scriptable.py
2022-05-18T03:53:22.3448905Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T03:53:22.3458053Z 
2022-05-18T03:53:22.3458254Z Running tests...
2022-05-18T03:53:22.3458626Z ----------------------------------------------------------------------
2022-05-18T03:53:22.6586653Z   test_backward_without_rpc (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14741
2022-05-18T03:53:22.6608391Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14742
2022-05-18T03:53:22.6631343Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14743
2022-05-18T03:53:22.6655252Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 14744
2022-05-18T03:53:23.2822379Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprvionulx
2022-05-18T03:53:23.2823704Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprvionulx/_remote_module_non_scriptable.py
2022-05-18T03:53:23.2902504Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8cg5h_7n
2022-05-18T03:53:23.2903801Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8cg5h_7n/_remote_module_non_scriptable.py
2022-05-18T03:53:23.3163649Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptlgdiu6l
2022-05-18T03:53:23.3164867Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptlgdiu6l/_remote_module_non_scriptable.py
2022-05-18T03:53:23.3417128Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0822kmio
2022-05-18T03:53:23.3418093Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0822kmio/_remote_module_non_scriptable.py
2022-05-18T03:53:23.5320576Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:53:23.5372156Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:53:23.5642131Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:53:23.5868408Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:53:23.9692234Z ok (1.623s)
2022-05-18T03:53:23.9692444Z 
2022-05-18T03:53:23.9693215Z ----------------------------------------------------------------------
2022-05-18T03:53:23.9693489Z Ran 1 test in 1.623s
2022-05-18T03:53:23.9693604Z 
2022-05-18T03:53:23.9693665Z OK
2022-05-18T03:53:23.9693806Z 
2022-05-18T03:53:23.9693902Z Generating XML reports...
2022-05-18T03:53:23.9728524Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220518035322.xml
2022-05-18T03:53:24.7386205Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpafqxje_u
2022-05-18T03:53:24.7387147Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpafqxje_u/_remote_module_non_scriptable.py
2022-05-18T03:53:24.9895625Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T03:53:24.9905750Z 
2022-05-18T03:53:24.9905876Z Running tests...
2022-05-18T03:53:24.9906279Z ----------------------------------------------------------------------
2022-05-18T03:53:25.3066591Z   test_backwards_nested_python_udf (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14980
2022-05-18T03:53:25.3089511Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14981
2022-05-18T03:53:25.3113031Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14982
2022-05-18T03:53:25.3137141Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 14983
2022-05-18T03:53:25.9198906Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxlw_43b4
2022-05-18T03:53:25.9199660Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxlw_43b4/_remote_module_non_scriptable.py
2022-05-18T03:53:25.9498396Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_3cm9obu
2022-05-18T03:53:25.9499925Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_3cm9obu/_remote_module_non_scriptable.py
2022-05-18T03:53:25.9618664Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprbbzl7r1
2022-05-18T03:53:25.9620083Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprbbzl7r1/_remote_module_non_scriptable.py
2022-05-18T03:53:25.9643530Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp28mxgpnk
2022-05-18T03:53:25.9645637Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp28mxgpnk/_remote_module_non_scriptable.py
2022-05-18T03:53:26.1665510Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:53:26.1968941Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:53:26.2079306Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:53:26.2144904Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:53:26.7177234Z ok (1.727s)
2022-05-18T03:53:26.7177438Z 
2022-05-18T03:53:26.7178177Z ----------------------------------------------------------------------
2022-05-18T03:53:26.7178462Z Ran 1 test in 1.727s
2022-05-18T03:53:26.7178578Z 
2022-05-18T03:53:26.7178643Z OK
2022-05-18T03:53:26.7178723Z 
2022-05-18T03:53:26.7178816Z Generating XML reports...
2022-05-18T03:53:26.7214468Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220518035324.xml
2022-05-18T03:53:27.4832436Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptb74sr0v
2022-05-18T03:53:27.4833785Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptb74sr0v/_remote_module_non_scriptable.py
2022-05-18T03:53:27.7351998Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T03:53:27.7361019Z 
2022-05-18T03:53:27.7361162Z Running tests...
2022-05-18T03:53:27.7361787Z ----------------------------------------------------------------------
2022-05-18T03:53:27.7375091Z   test_clean_context_during_backward (__main__.TensorPipeDistAutogradTest)
2022-05-18T03:53:28.0474728Z This test simulates the situation where the 'backward' call might throw ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15219
2022-05-18T03:53:28.0497570Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15220
2022-05-18T03:53:28.0520663Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15221
2022-05-18T03:53:28.0544921Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 15222
2022-05-18T03:53:28.6653805Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjohoklz_
2022-05-18T03:53:28.6654600Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjohoklz_/_remote_module_non_scriptable.py
2022-05-18T03:53:28.6760334Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjw_vofqx
2022-05-18T03:53:28.6761157Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjw_vofqx/_remote_module_non_scriptable.py
2022-05-18T03:53:28.6831155Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqx0zdp76
2022-05-18T03:53:28.6833035Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqx0zdp76/_remote_module_non_scriptable.py
2022-05-18T03:53:28.6927897Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbmke2c7_
2022-05-18T03:53:28.6930001Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbmke2c7_/_remote_module_non_scriptable.py
2022-05-18T03:53:28.9127924Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:53:28.9218308Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:53:28.9314076Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:53:28.9399399Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:53:29.1957178Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:53:29.2058619Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:53:29.2059466Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:53:29.2061246Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:53:29.2063178Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:53:29.2163474Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:53:29.2164486Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:53:29.2165604Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:53:29.4011029Z [W tensorpipe_agent.cpp:728] RPC agent for worker2 encountered error when reading incoming request from worker0: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2022-05-18T03:53:29.4024196Z [W tensorpipe_agent.cpp:728] RPC agent for worker0 encountered error when reading incoming request from worker3: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2022-05-18T03:53:29.4025188Z [W tensorpipe_agent.cpp:728] RPC agent for worker0 encountered error when reading incoming request from worker2: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2022-05-18T03:53:29.4026319Z [W tensorpipe_agent.cpp:728] RPC agent for worker0 encountered error when reading incoming request from worker1: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2022-05-18T03:53:29.4069748Z [W tensorpipe_agent.cpp:728] RPC agent for worker2 encountered error when reading incoming request from worker3: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2022-05-18T03:53:29.4070778Z [W tensorpipe_agent.cpp:728] RPC agent for worker2 encountered error when reading incoming request from worker1: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2022-05-18T03:53:29.4112526Z [W tensorpipe_agent.cpp:728] RPC agent for worker3 encountered error when reading incoming request from worker1: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2022-05-18T03:53:29.4113527Z [W tensorpipe_agent.cpp:728] RPC agent for worker3 encountered error when reading incoming request from worker0: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2022-05-18T03:53:29.4114099Z [W tensorpipe_agent.cpp:728] RPC agent for worker3 encountered error when reading incoming request from worker2: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2022-05-18T03:53:29.4155924Z [W tensorpipe_agent.cpp:728] RPC agent for worker1 encountered error when reading incoming request from worker0: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2022-05-18T03:53:29.4156710Z [W tensorpipe_agent.cpp:728] RPC agent for worker1 encountered error when reading incoming request from worker3: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2022-05-18T03:53:29.4157478Z [W tensorpipe_agent.cpp:728] RPC agent for worker1 encountered error when reading incoming request from worker2: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2022-05-18T03:53:29.6588908Z ok (1.922s)
2022-05-18T03:53:29.6589133Z 
2022-05-18T03:53:29.6589499Z ----------------------------------------------------------------------
2022-05-18T03:53:29.6589773Z Ran 1 test in 1.923s
2022-05-18T03:53:29.6589874Z 
2022-05-18T03:53:29.6589934Z OK
2022-05-18T03:53:29.6590029Z 
2022-05-18T03:53:29.6590128Z Generating XML reports...
2022-05-18T03:53:29.6626828Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220518035327.xml
2022-05-18T03:53:30.4261291Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzpygqpue
2022-05-18T03:53:30.4261883Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzpygqpue/_remote_module_non_scriptable.py
2022-05-18T03:53:30.6791932Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T03:53:30.6801761Z 
2022-05-18T03:53:30.6801895Z Running tests...
2022-05-18T03:53:30.6802809Z ----------------------------------------------------------------------
2022-05-18T03:53:30.9911818Z   test_context_cleanup_nested_rpc (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15454
2022-05-18T03:53:30.9935228Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15455
2022-05-18T03:53:30.9957581Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15456
2022-05-18T03:53:30.9981906Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 15457
2022-05-18T03:53:31.6090389Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7c06nnuy
2022-05-18T03:53:31.6093016Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7c06nnuy/_remote_module_non_scriptable.py
2022-05-18T03:53:31.6245568Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdzxne6bh
2022-05-18T03:53:31.6246489Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdzxne6bh/_remote_module_non_scriptable.py
2022-05-18T03:53:31.6468346Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9fb6wi8k
2022-05-18T03:53:31.6469361Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9fb6wi8k/_remote_module_non_scriptable.py
2022-05-18T03:53:31.6621599Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd86db1c4
2022-05-18T03:53:31.6622840Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd86db1c4/_remote_module_non_scriptable.py
2022-05-18T03:53:31.8548767Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:53:31.8694726Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:53:31.8940544Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:53:31.9099155Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:53:32.1412866Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:53:32.1512561Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:53:32.1513785Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:53:32.1517335Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:53:32.1518671Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:53:32.1618183Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:53:32.1619655Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:53:32.1620936Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:53:32.5023497Z ok (1.822s)
2022-05-18T03:53:32.5024028Z 
2022-05-18T03:53:32.5024603Z ----------------------------------------------------------------------
2022-05-18T03:53:32.5025074Z Ran 1 test in 1.822s
2022-05-18T03:53:32.5025304Z 
2022-05-18T03:53:32.5025372Z OK
2022-05-18T03:53:32.5025466Z 
2022-05-18T03:53:32.5025547Z Generating XML reports...
2022-05-18T03:53:32.5058667Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220518035330.xml
2022-05-18T03:53:33.2670646Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_bk2in_l
2022-05-18T03:53:33.2671754Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_bk2in_l/_remote_module_non_scriptable.py
2022-05-18T03:53:33.5207043Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T03:53:33.5216301Z 
2022-05-18T03:53:33.5216438Z Running tests...
2022-05-18T03:53:33.5216914Z ----------------------------------------------------------------------
2022-05-18T03:53:33.8332162Z   test_context_cleanup_no_tensors (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15685
2022-05-18T03:53:33.8354662Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15686
2022-05-18T03:53:33.8377139Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15687
2022-05-18T03:53:33.8401338Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 15688
2022-05-18T03:53:34.4558626Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp06h0j2jd
2022-05-18T03:53:34.4559817Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp06h0j2jd/_remote_module_non_scriptable.py
2022-05-18T03:53:34.4819707Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppv695f6c
2022-05-18T03:53:34.4820808Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppv695f6c/_remote_module_non_scriptable.py
2022-05-18T03:53:34.5024271Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp77tv84nc
2022-05-18T03:53:34.5026838Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp77tv84nc/_remote_module_non_scriptable.py
2022-05-18T03:53:34.5173579Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1e4l5xx9
2022-05-18T03:53:34.5175058Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1e4l5xx9/_remote_module_non_scriptable.py
2022-05-18T03:53:34.7027616Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:53:34.7298437Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:53:34.7477648Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:53:34.7642569Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:53:34.9828868Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:53:35.0028966Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:53:35.0030225Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:53:35.0032277Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:53:35.0034791Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:53:35.0036191Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:53:35.0133256Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:53:35.0134299Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:53:35.3443773Z ok (1.822s)
2022-05-18T03:53:35.3443937Z 
2022-05-18T03:53:35.3444335Z ----------------------------------------------------------------------
2022-05-18T03:53:35.3444786Z Ran 1 test in 1.823s
2022-05-18T03:53:35.3444982Z 
2022-05-18T03:53:35.3445102Z OK
2022-05-18T03:53:35.3445248Z 
2022-05-18T03:53:35.3445342Z Generating XML reports...
2022-05-18T03:53:35.3478851Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220518035333.xml
2022-05-18T03:53:36.1139006Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7uzn7aw6
2022-05-18T03:53:36.1139757Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7uzn7aw6/_remote_module_non_scriptable.py
2022-05-18T03:53:36.3669159Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T03:53:36.3678528Z 
2022-05-18T03:53:36.3678663Z Running tests...
2022-05-18T03:53:36.3679228Z ----------------------------------------------------------------------
2022-05-18T03:53:36.6833340Z   test_context_cleanup_tensor_no_grad (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15916
2022-05-18T03:53:36.6856749Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15917
2022-05-18T03:53:36.6879866Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15918
2022-05-18T03:53:36.6904233Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 15919
2022-05-18T03:53:37.3612178Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpryenh4ag
2022-05-18T03:53:37.3612973Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpryenh4ag/_remote_module_non_scriptable.py
2022-05-18T03:53:37.4096524Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2mza9qg2
2022-05-18T03:53:37.4097419Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2mza9qg2/_remote_module_non_scriptable.py
2022-05-18T03:53:37.4462251Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1gknxxss
2022-05-18T03:53:37.4463730Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1gknxxss/_remote_module_non_scriptable.py
2022-05-18T03:53:37.5528042Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt0y4l1m_
2022-05-18T03:53:37.5529050Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt0y4l1m_/_remote_module_non_scriptable.py
2022-05-18T03:53:37.6094671Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:53:37.6564295Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:53:37.7583027Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:53:37.8205993Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:53:38.0474041Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:53:38.0573882Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:53:38.0681712Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:53:38.0682932Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:53:38.0684292Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:53:38.0685277Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:53:38.0686339Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:53:38.0687806Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:53:38.3947764Z ok (2.027s)
2022-05-18T03:53:38.3947995Z 
2022-05-18T03:53:38.3948366Z ----------------------------------------------------------------------
2022-05-18T03:53:38.3948643Z Ran 1 test in 2.027s
2022-05-18T03:53:38.3948784Z 
2022-05-18T03:53:38.3949357Z OK
2022-05-18T03:53:38.3949451Z 
2022-05-18T03:53:38.3950026Z Generating XML reports...
2022-05-18T03:53:38.3986831Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220518035336.xml
2022-05-18T03:53:39.1572373Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0cxth85y
2022-05-18T03:53:39.1573386Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0cxth85y/_remote_module_non_scriptable.py
2022-05-18T03:53:39.4082863Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T03:53:39.4092432Z 
2022-05-18T03:53:39.4092739Z Running tests...
2022-05-18T03:53:39.4093350Z ----------------------------------------------------------------------
2022-05-18T03:53:39.7261198Z   test_context_cleanup_tensor_with_grad (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16147
2022-05-18T03:53:39.7284454Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16148
2022-05-18T03:53:39.7308059Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16149
2022-05-18T03:53:39.7332066Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 16150
2022-05-18T03:53:40.4220686Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0d6mx66d
2022-05-18T03:53:40.4221794Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0d6mx66d/_remote_module_non_scriptable.py
2022-05-18T03:53:40.4237056Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnmptemwp
2022-05-18T03:53:40.4239173Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnmptemwp/_remote_module_non_scriptable.py
2022-05-18T03:53:40.4612569Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0vjssp71
2022-05-18T03:53:40.4613689Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0vjssp71/_remote_module_non_scriptable.py
2022-05-18T03:53:40.4644648Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp2ilpdt7
2022-05-18T03:53:40.4646203Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp2ilpdt7/_remote_module_non_scriptable.py
2022-05-18T03:53:40.6703814Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:53:40.6704962Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:53:40.7103559Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:53:40.7112656Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:53:40.9774610Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:53:40.9875540Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:53:40.9877022Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:53:40.9878329Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:53:40.9879209Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:53:40.9880284Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:53:40.9881457Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:53:40.9882606Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:53:41.4377850Z ok (2.028s)
2022-05-18T03:53:41.4378392Z 
2022-05-18T03:53:41.4379023Z ----------------------------------------------------------------------
2022-05-18T03:53:41.4379525Z Ran 1 test in 2.029s
2022-05-18T03:53:41.4379753Z 
2022-05-18T03:53:41.4379850Z OK
2022-05-18T03:53:41.4379985Z 
2022-05-18T03:53:41.4380083Z Generating XML reports...
2022-05-18T03:53:41.4413331Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220518035339.xml
2022-05-18T03:53:42.2060832Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6_h325xg
2022-05-18T03:53:42.2061716Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6_h325xg/_remote_module_non_scriptable.py
2022-05-18T03:53:42.4614150Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T03:53:42.4624535Z 
2022-05-18T03:53:42.4624624Z Running tests...
2022-05-18T03:53:42.4625144Z ----------------------------------------------------------------------
2022-05-18T03:53:42.7812811Z   test_debug_info (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16378
2022-05-18T03:53:42.7836512Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16379
2022-05-18T03:53:42.7859740Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16380
2022-05-18T03:53:42.7884316Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 16381
2022-05-18T03:53:43.4084638Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy8772y3z
2022-05-18T03:53:43.4085654Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy8772y3z/_remote_module_non_scriptable.py
2022-05-18T03:53:43.4088339Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb8oyo3o4
2022-05-18T03:53:43.4089480Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb8oyo3o4/_remote_module_non_scriptable.py
2022-05-18T03:53:43.4102031Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq9fb52db
2022-05-18T03:53:43.4104167Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq9fb52db/_remote_module_non_scriptable.py
2022-05-18T03:53:43.4635500Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu2_yqfum
2022-05-18T03:53:43.4636502Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu2_yqfum/_remote_module_non_scriptable.py
2022-05-18T03:53:43.6549659Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:53:43.6565631Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:53:43.6573941Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:53:43.7108662Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:53:43.9613324Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:53:43.9616434Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:53:43.9617993Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:53:43.9620556Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:53:43.9623621Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:53:43.9624856Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:53:43.9717061Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:53:43.9718326Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:53:44.3928486Z ok (1.930s)
2022-05-18T03:53:44.3928773Z 
2022-05-18T03:53:44.3929226Z ----------------------------------------------------------------------
2022-05-18T03:53:44.3929481Z Ran 1 test in 1.930s
2022-05-18T03:53:44.3929583Z 
2022-05-18T03:53:44.3929646Z OK
2022-05-18T03:53:44.3929739Z 
2022-05-18T03:53:44.3929833Z Generating XML reports...
2022-05-18T03:53:44.3963207Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220518035342.xml
2022-05-18T03:53:45.1650057Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdtgq4zax
2022-05-18T03:53:45.1650611Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdtgq4zax/_remote_module_non_scriptable.py
2022-05-18T03:53:45.4168061Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T03:53:45.4177354Z 
2022-05-18T03:53:45.4177533Z Running tests...
2022-05-18T03:53:45.4177986Z ----------------------------------------------------------------------
2022-05-18T03:53:45.7252514Z   test_dist_autograd_profiling (__main__.TensorPipeDistAutogradTest) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77318 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure IN_CI is not set and you are not using the flag --import-disabled-tests. (0.307s)
2022-05-18T03:53:45.7253021Z 
2022-05-18T03:53:45.7253244Z ----------------------------------------------------------------------
2022-05-18T03:53:45.7253699Z Ran 1 test in 0.307s
2022-05-18T03:53:45.7253815Z 
2022-05-18T03:53:45.7253887Z OK (skipped=1)
2022-05-18T03:53:45.7253994Z 
2022-05-18T03:53:45.7254125Z Generating XML reports...
2022-05-18T03:53:45.7278064Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220518035345.xml
2022-05-18T03:53:46.4377296Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu1qbv2ue
2022-05-18T03:53:46.4378084Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu1qbv2ue/_remote_module_non_scriptable.py
2022-05-18T03:53:46.6896836Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T03:53:46.6906088Z 
2022-05-18T03:53:46.6906178Z Running tests...
2022-05-18T03:53:46.6906863Z ----------------------------------------------------------------------
2022-05-18T03:53:47.0013640Z   test_error_in_context (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16639
2022-05-18T03:53:47.0036530Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16640
2022-05-18T03:53:47.0059577Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16641
2022-05-18T03:53:47.0083558Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 16642
2022-05-18T03:53:47.6695956Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyqbk_6p1
2022-05-18T03:53:47.6696760Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyqbk_6p1/_remote_module_non_scriptable.py
2022-05-18T03:53:47.7012237Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi5saqjek
2022-05-18T03:53:47.7013141Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi5saqjek/_remote_module_non_scriptable.py
2022-05-18T03:53:47.7085905Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpaffl1kx8
2022-05-18T03:53:47.7087070Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpaffl1kx8/_remote_module_non_scriptable.py
2022-05-18T03:53:47.7281422Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc03est_f
2022-05-18T03:53:47.7282721Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc03est_f/_remote_module_non_scriptable.py
2022-05-18T03:53:47.9163849Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:53:47.9486530Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:53:47.9537740Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:53:47.9746861Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:53:48.4124922Z ok (1.722s)
2022-05-18T03:53:48.4125189Z 
2022-05-18T03:53:48.4125671Z ----------------------------------------------------------------------
2022-05-18T03:53:48.4125959Z Ran 1 test in 1.722s
2022-05-18T03:53:48.4126175Z 
2022-05-18T03:53:48.4126278Z OK
2022-05-18T03:53:48.4126457Z 
2022-05-18T03:53:48.4126636Z Generating XML reports...
2022-05-18T03:53:48.4161070Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220518035346.xml
2022-05-18T03:53:49.1855022Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpaw7zrfn2
2022-05-18T03:53:49.1855802Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpaw7zrfn2/_remote_module_non_scriptable.py
2022-05-18T03:53:49.4379472Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T03:53:49.4388391Z 
2022-05-18T03:53:49.4388528Z Running tests...
2022-05-18T03:53:49.4389150Z ----------------------------------------------------------------------
2022-05-18T03:53:49.7499471Z   test_grad_copy_sparse_indices_extra_ref (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16858
2022-05-18T03:53:49.7522617Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16859
2022-05-18T03:53:49.7545439Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16860
2022-05-18T03:53:49.7569157Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 16861
2022-05-18T03:53:50.3622098Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprq5seyam
2022-05-18T03:53:50.3623027Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprq5seyam/_remote_module_non_scriptable.py
2022-05-18T03:53:50.3734212Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp77_baq0j
2022-05-18T03:53:50.3735391Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp77_baq0j/_remote_module_non_scriptable.py
2022-05-18T03:53:50.3966766Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcps8v2gt
2022-05-18T03:53:50.3967476Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdxpjm6rt
2022-05-18T03:53:50.3968203Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcps8v2gt/_remote_module_non_scriptable.py
2022-05-18T03:53:50.3968945Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdxpjm6rt/_remote_module_non_scriptable.py
2022-05-18T03:53:50.6097192Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:53:50.6209140Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:53:50.6441018Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:53:50.6464919Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:53:50.8406620Z /opt/conda/lib/python3.7/site-packages/torch/nn/functional.py:2263: UserWarning: Argument order of nn.functional.embedding_bag was changed. Usage `embedding_bag(weight, input, ...)` is deprecated, and should now be `embedding_bag(input, weight, ...)`.
2022-05-18T03:53:50.8407892Z   "Argument order of nn.functional.embedding_bag was changed. "
2022-05-18T03:53:50.8610251Z /opt/conda/lib/python3.7/site-packages/torch/nn/functional.py:2263: UserWarning: Argument order of nn.functional.embedding_bag was changed. Usage `embedding_bag(weight, input, ...)` is deprecated, and should now be `embedding_bag(input, weight, ...)`.
2022-05-18T03:53:50.8611050Z   "Argument order of nn.functional.embedding_bag was changed. "
2022-05-18T03:53:50.8612058Z /opt/conda/lib/python3.7/site-packages/torch/nn/functional.py:2263: UserWarning: Argument order of nn.functional.embedding_bag was changed. Usage `embedding_bag(weight, input, ...)` is deprecated, and should now be `embedding_bag(input, weight, ...)`.
2022-05-18T03:53:50.8612799Z   "Argument order of nn.functional.embedding_bag was changed. "
2022-05-18T03:53:50.8613920Z /opt/conda/lib/python3.7/site-packages/torch/nn/functional.py:2263: UserWarning: Argument order of nn.functional.embedding_bag was changed. Usage `embedding_bag(weight, input, ...)` is deprecated, and should now be `embedding_bag(input, weight, ...)`.
2022-05-18T03:53:50.8614650Z   "Argument order of nn.functional.embedding_bag was changed. "
2022-05-18T03:53:51.1609200Z ok (1.722s)
2022-05-18T03:53:51.1609550Z 
2022-05-18T03:53:51.1610056Z ----------------------------------------------------------------------
2022-05-18T03:53:51.1610497Z Ran 1 test in 1.722s
2022-05-18T03:53:51.1610638Z 
2022-05-18T03:53:51.1610702Z OK
2022-05-18T03:53:51.1610795Z 
2022-05-18T03:53:51.1610889Z Generating XML reports...
2022-05-18T03:53:51.1643891Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220518035349.xml
2022-05-18T03:53:51.9255076Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw3au16i6
2022-05-18T03:53:51.9256272Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw3au16i6/_remote_module_non_scriptable.py
2022-05-18T03:53:52.1761125Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T03:53:52.1770352Z 
2022-05-18T03:53:52.1770785Z Running tests...
2022-05-18T03:53:52.1771195Z ----------------------------------------------------------------------
2022-05-18T03:53:52.4920512Z   test_grad_only_on_return_value (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17109
2022-05-18T03:53:52.4943621Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17110
2022-05-18T03:53:52.4967463Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17111
2022-05-18T03:53:52.4991351Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 17112
2022-05-18T03:53:53.1191680Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxrou2sz_
2022-05-18T03:53:53.1192797Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxrou2sz_/_remote_module_non_scriptable.py
2022-05-18T03:53:53.1322484Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl74yeivb
2022-05-18T03:53:53.1323386Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl74yeivb/_remote_module_non_scriptable.py
2022-05-18T03:53:53.1455994Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnfps_hqj
2022-05-18T03:53:53.1457230Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnfps_hqj/_remote_module_non_scriptable.py
2022-05-18T03:53:53.1667169Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwm1r2jc0
2022-05-18T03:53:53.1668390Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwm1r2jc0/_remote_module_non_scriptable.py
2022-05-18T03:53:53.3661155Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:53:53.3797311Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:53:53.3914441Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:53:53.4138446Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:53:53.6550074Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:53:53.6749207Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:53:53.6750812Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:53:53.6751902Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:53:53.6752469Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:53:53.6753228Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:53:53.6754466Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:53:53.6755295Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:53:54.1036178Z ok (1.926s)
2022-05-18T03:53:54.1036392Z 
2022-05-18T03:53:54.1036744Z ----------------------------------------------------------------------
2022-05-18T03:53:54.1037003Z Ran 1 test in 1.926s
2022-05-18T03:53:54.1037121Z 
2022-05-18T03:53:54.1037189Z OK
2022-05-18T03:53:54.1037324Z 
2022-05-18T03:53:54.1037425Z Generating XML reports...
2022-05-18T03:53:54.1070947Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220518035352.xml
2022-05-18T03:53:54.8664838Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5sxyeead
2022-05-18T03:53:54.8666136Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5sxyeead/_remote_module_non_scriptable.py
2022-05-18T03:53:55.1205600Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T03:53:55.1215072Z 
2022-05-18T03:53:55.1215209Z Running tests...
2022-05-18T03:53:55.1215807Z ----------------------------------------------------------------------
2022-05-18T03:53:55.4415966Z   test_grad_only_on_return_value_remote (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17360
2022-05-18T03:53:55.4439177Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17361
2022-05-18T03:53:55.4463427Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17362
2022-05-18T03:53:55.4487783Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 17363
2022-05-18T03:53:56.1080968Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2j80b0yx
2022-05-18T03:53:56.1081762Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2j80b0yx/_remote_module_non_scriptable.py
2022-05-18T03:53:56.1286767Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6zaetfsd
2022-05-18T03:53:56.1287547Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6zaetfsd/_remote_module_non_scriptable.py
2022-05-18T03:53:56.1455807Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph43lsh9k
2022-05-18T03:53:56.1457020Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph43lsh9k/_remote_module_non_scriptable.py
2022-05-18T03:53:56.1800234Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf_457y3l
2022-05-18T03:53:56.1801048Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf_457y3l/_remote_module_non_scriptable.py
2022-05-18T03:53:56.3578224Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:53:56.3764268Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:53:56.3933915Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:53:56.4268890Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:53:56.6373657Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:53:56.6374629Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:53:56.6475008Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:53:56.6475873Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:53:56.6477158Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:53:56.6478334Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:53:56.6479934Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:53:56.6483278Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:53:57.0533087Z ok (1.932s)
2022-05-18T03:53:57.0533285Z 
2022-05-18T03:53:57.0533756Z ----------------------------------------------------------------------
2022-05-18T03:53:57.0534212Z Ran 1 test in 1.932s
2022-05-18T03:53:57.0534419Z 
2022-05-18T03:53:57.0534530Z OK
2022-05-18T03:53:57.0534679Z 
2022-05-18T03:53:57.0534772Z Generating XML reports...
2022-05-18T03:53:57.0568922Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220518035355.xml
2022-05-18T03:53:57.8239656Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpoz2_hne2
2022-05-18T03:53:57.8240409Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpoz2_hne2/_remote_module_non_scriptable.py
2022-05-18T03:53:58.0773478Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T03:53:58.0784137Z 
2022-05-18T03:53:58.0784446Z Running tests...
2022-05-18T03:53:58.0785146Z ----------------------------------------------------------------------
2022-05-18T03:53:58.3927187Z   test_graph_for_builtin_call (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17611
2022-05-18T03:53:58.3949226Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17612
2022-05-18T03:53:58.3972959Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17613
2022-05-18T03:53:58.3998410Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 17614
2022-05-18T03:53:59.0552530Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbx7xwn3n
2022-05-18T03:53:59.0553644Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbx7xwn3n/_remote_module_non_scriptable.py
2022-05-18T03:53:59.0980780Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyyqw16zw
2022-05-18T03:53:59.0982221Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyyqw16zw/_remote_module_non_scriptable.py
2022-05-18T03:53:59.1285295Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptdaao4mg
2022-05-18T03:53:59.1286119Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptdaao4mg/_remote_module_non_scriptable.py
2022-05-18T03:53:59.1418016Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxs68ejt1
2022-05-18T03:53:59.1419243Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxs68ejt1/_remote_module_non_scriptable.py
2022-05-18T03:53:59.3030094Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:53:59.3441462Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:53:59.3762215Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:53:59.3885450Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:53:59.6133162Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:53:59.6134033Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:53:59.6234016Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:53:59.6236662Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:53:59.6237808Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:53:59.6239416Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:53:59.6240587Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:53:59.6241735Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:54:00.0041019Z ok (1.925s)
2022-05-18T03:54:00.0041254Z 
2022-05-18T03:54:00.0041786Z ----------------------------------------------------------------------
2022-05-18T03:54:00.0042190Z Ran 1 test in 1.926s
2022-05-18T03:54:00.0042540Z 
2022-05-18T03:54:00.0042603Z OK
2022-05-18T03:54:00.0042696Z 
2022-05-18T03:54:00.0042793Z Generating XML reports...
2022-05-18T03:54:00.0076282Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220518035358.xml
2022-05-18T03:54:00.7736793Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwvh0v0ly
2022-05-18T03:54:00.7737863Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwvh0v0ly/_remote_module_non_scriptable.py
2022-05-18T03:54:01.0267023Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T03:54:01.0276695Z 
2022-05-18T03:54:01.0276800Z Running tests...
2022-05-18T03:54:01.0277472Z ----------------------------------------------------------------------
2022-05-18T03:54:01.3429506Z   test_graph_for_builtin_remote_call (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17842
2022-05-18T03:54:01.3452095Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17843
2022-05-18T03:54:01.3475103Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17844
2022-05-18T03:54:01.3498654Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 17845
2022-05-18T03:54:01.9962143Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmlac4438
2022-05-18T03:54:01.9962944Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmlac4438/_remote_module_non_scriptable.py
2022-05-18T03:54:02.0101827Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpij5wumh_
2022-05-18T03:54:02.0103026Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpij5wumh_/_remote_module_non_scriptable.py
2022-05-18T03:54:02.0233171Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6awph5i9
2022-05-18T03:54:02.0234203Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6awph5i9/_remote_module_non_scriptable.py
2022-05-18T03:54:02.0455217Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpspjyv2fs
2022-05-18T03:54:02.0456249Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpspjyv2fs/_remote_module_non_scriptable.py
2022-05-18T03:54:02.2429945Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:54:02.2584146Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:54:02.2727593Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:54:02.2931231Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:54:02.5272279Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:54:02.5372476Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:54:02.5473481Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:54:02.5478783Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:54:02.5481830Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:54:02.5483440Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:54:02.5484846Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:54:02.5486259Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:54:02.9542484Z ok (1.926s)
2022-05-18T03:54:02.9543078Z 
2022-05-18T03:54:02.9543623Z ----------------------------------------------------------------------
2022-05-18T03:54:02.9544415Z Ran 1 test in 1.927s
2022-05-18T03:54:02.9544551Z 
2022-05-18T03:54:02.9544599Z OK
2022-05-18T03:54:02.9544693Z 
2022-05-18T03:54:02.9544849Z Generating XML reports...
2022-05-18T03:54:02.9579746Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220518035401.xml
2022-05-18T03:54:03.7185943Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9br7ehyz
2022-05-18T03:54:03.7186573Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9br7ehyz/_remote_module_non_scriptable.py
2022-05-18T03:54:03.9728627Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T03:54:03.9738570Z 
2022-05-18T03:54:03.9738660Z Running tests...
2022-05-18T03:54:03.9739598Z ----------------------------------------------------------------------
2022-05-18T03:54:04.2871200Z   test_graph_for_py_nested_call (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18073
2022-05-18T03:54:04.2894638Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18074
2022-05-18T03:54:04.2917534Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18075
2022-05-18T03:54:04.2941398Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 18076
2022-05-18T03:54:04.9812698Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwld96c64
2022-05-18T03:54:04.9813452Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwld96c64/_remote_module_non_scriptable.py
2022-05-18T03:54:04.9856196Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4u9ghdow
2022-05-18T03:54:04.9857500Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4u9ghdow/_remote_module_non_scriptable.py
2022-05-18T03:54:04.9910007Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvlb283n4
2022-05-18T03:54:04.9911467Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvlb283n4/_remote_module_non_scriptable.py
2022-05-18T03:54:05.0523831Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn4cnzv54
2022-05-18T03:54:05.0524735Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn4cnzv54/_remote_module_non_scriptable.py
2022-05-18T03:54:05.2297318Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:54:05.2329422Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:54:05.2362216Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:54:05.3018689Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:54:05.5292336Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:54:05.5493914Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:54:05.5598236Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:54:05.5601997Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:54:05.5603276Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:54:05.5604440Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:54:05.5605451Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:54:05.5609271Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:54:05.8983943Z ok (1.924s)
2022-05-18T03:54:05.8984180Z 
2022-05-18T03:54:05.8984565Z ----------------------------------------------------------------------
2022-05-18T03:54:05.8985035Z Ran 1 test in 1.924s
2022-05-18T03:54:05.8985153Z 
2022-05-18T03:54:05.8985221Z OK
2022-05-18T03:54:05.8985316Z 
2022-05-18T03:54:05.8985412Z Generating XML reports...
2022-05-18T03:54:05.9018704Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220518035403.xml
2022-05-18T03:54:06.6811616Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyrzcviiw
2022-05-18T03:54:06.6812714Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyrzcviiw/_remote_module_non_scriptable.py
2022-05-18T03:54:06.9344014Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T03:54:06.9353988Z 
2022-05-18T03:54:06.9354285Z Running tests...
2022-05-18T03:54:06.9354933Z ----------------------------------------------------------------------
2022-05-18T03:54:07.2480667Z   test_graph_for_py_nested_call_itself (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18304
2022-05-18T03:54:07.2504033Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18305
2022-05-18T03:54:07.2526570Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18306
2022-05-18T03:54:07.2551245Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 18307
2022-05-18T03:54:07.9381044Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgiapaot1
2022-05-18T03:54:07.9381787Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgiapaot1/_remote_module_non_scriptable.py
2022-05-18T03:54:07.9459774Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2qdu2rbn
2022-05-18T03:54:07.9461113Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2qdu2rbn/_remote_module_non_scriptable.py
2022-05-18T03:54:07.9670544Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp39esbdws
2022-05-18T03:54:07.9671443Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp39esbdws/_remote_module_non_scriptable.py
2022-05-18T03:54:07.9828459Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwhkeb50c
2022-05-18T03:54:07.9829708Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwhkeb50c/_remote_module_non_scriptable.py
2022-05-18T03:54:08.1847184Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:54:08.1930861Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:54:08.2143859Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:54:08.2294376Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:54:08.4614640Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:54:08.4713997Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:54:08.4815918Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:54:08.4819442Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:54:08.4820900Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:54:08.4823460Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:54:08.4824561Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:54:08.4825764Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:54:08.8597276Z ok (1.924s)
2022-05-18T03:54:08.8597566Z 
2022-05-18T03:54:08.8598400Z ----------------------------------------------------------------------
2022-05-18T03:54:08.8598663Z Ran 1 test in 1.924s
2022-05-18T03:54:08.8598778Z 
2022-05-18T03:54:08.8598841Z OK
2022-05-18T03:54:08.8598919Z 
2022-05-18T03:54:08.8599014Z Generating XML reports...
2022-05-18T03:54:08.8632719Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220518035406.xml
2022-05-18T03:54:09.6226703Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjjpe5yr4
2022-05-18T03:54:09.6227182Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjjpe5yr4/_remote_module_non_scriptable.py
2022-05-18T03:54:09.8744118Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T03:54:09.8754059Z 
2022-05-18T03:54:09.8754157Z Running tests...
2022-05-18T03:54:09.8754588Z ----------------------------------------------------------------------
2022-05-18T03:54:10.1884719Z   test_graph_for_py_nested_remote_call (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18535
2022-05-18T03:54:10.1907984Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18536
2022-05-18T03:54:10.1931058Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18537
2022-05-18T03:54:10.1956188Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 18538
2022-05-18T03:54:10.8281382Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpaavfvbzb
2022-05-18T03:54:10.8282154Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpaavfvbzb/_remote_module_non_scriptable.py
2022-05-18T03:54:10.8784140Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpymx7_wbi
2022-05-18T03:54:10.8785365Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpymx7_wbi/_remote_module_non_scriptable.py
2022-05-18T03:54:10.9078756Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfpl22del
2022-05-18T03:54:10.9079526Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfpl22del/_remote_module_non_scriptable.py
2022-05-18T03:54:10.9132707Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2_lhtq68
2022-05-18T03:54:10.9134316Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2_lhtq68/_remote_module_non_scriptable.py
2022-05-18T03:54:11.0793595Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:54:11.1258321Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:54:11.1535542Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:54:11.1607428Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:54:11.3884051Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:54:11.3885044Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:54:11.3885860Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:54:11.3887122Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:54:11.3887989Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:54:11.3889105Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:54:11.3889956Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:54:11.3891163Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:54:11.7999043Z ok (1.924s)
2022-05-18T03:54:11.7999315Z 
2022-05-18T03:54:11.7999822Z ----------------------------------------------------------------------
2022-05-18T03:54:11.8000279Z Ran 1 test in 1.924s
2022-05-18T03:54:11.8000490Z 
2022-05-18T03:54:11.8000593Z OK
2022-05-18T03:54:11.8000723Z 
2022-05-18T03:54:11.8000818Z Generating XML reports...
2022-05-18T03:54:11.8034245Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220518035409.xml
2022-05-18T03:54:12.5657387Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf88glz57
2022-05-18T03:54:12.5657846Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf88glz57/_remote_module_non_scriptable.py
2022-05-18T03:54:12.8181185Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T03:54:12.8191239Z 
2022-05-18T03:54:12.8191579Z Running tests...
2022-05-18T03:54:12.8191971Z ----------------------------------------------------------------------
2022-05-18T03:54:13.1301405Z   test_graph_for_py_nested_remote_call_itself (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18766
2022-05-18T03:54:13.1324712Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18767
2022-05-18T03:54:13.1348295Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18768
2022-05-18T03:54:13.1372199Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 18769
2022-05-18T03:54:13.7265995Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpleftq03v
2022-05-18T03:54:13.7267204Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpleftq03v/_remote_module_non_scriptable.py
2022-05-18T03:54:13.7345694Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz2on0vxz
2022-05-18T03:54:13.7346746Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz2on0vxz/_remote_module_non_scriptable.py
2022-05-18T03:54:13.7813405Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxpokv81m
2022-05-18T03:54:13.7814163Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxpokv81m/_remote_module_non_scriptable.py
2022-05-18T03:54:13.7898757Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjdrbj9nq
2022-05-18T03:54:13.7900070Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjdrbj9nq/_remote_module_non_scriptable.py
2022-05-18T03:54:13.9754921Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:54:13.9816464Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:54:14.0278544Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:54:14.0386592Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:54:14.3013480Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:54:14.3014403Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:54:14.3112430Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:54:14.3117007Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:54:14.3118820Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:54:14.3120014Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:54:14.3121526Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:54:14.3216163Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:54:14.6413048Z ok (1.822s)
2022-05-18T03:54:14.6413332Z 
2022-05-18T03:54:14.6413723Z ----------------------------------------------------------------------
2022-05-18T03:54:14.6413989Z Ran 1 test in 1.822s
2022-05-18T03:54:14.6414106Z 
2022-05-18T03:54:14.6414173Z OK
2022-05-18T03:54:14.6414251Z 
2022-05-18T03:54:14.6414346Z Generating XML reports...
2022-05-18T03:54:14.6450698Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220518035412.xml
2022-05-18T03:54:15.4254205Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcrhrd6s5
2022-05-18T03:54:15.4255203Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcrhrd6s5/_remote_module_non_scriptable.py
2022-05-18T03:54:15.6800956Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T03:54:15.6810805Z 
2022-05-18T03:54:15.6810916Z Running tests...
2022-05-18T03:54:15.6811411Z ----------------------------------------------------------------------
2022-05-18T03:54:15.9958868Z   test_graph_for_python_call (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18997
2022-05-18T03:54:15.9980755Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18998
2022-05-18T03:54:16.0003603Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18999
2022-05-18T03:54:16.0027805Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 19000
2022-05-18T03:54:16.6236146Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9_bxo1pp
2022-05-18T03:54:16.6236900Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9_bxo1pp/_remote_module_non_scriptable.py
2022-05-18T03:54:16.6907831Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7dc04uy6
2022-05-18T03:54:16.6909116Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7dc04uy6/_remote_module_non_scriptable.py
2022-05-18T03:54:16.7220038Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6ng_rhu1
2022-05-18T03:54:16.7220823Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6ng_rhu1/_remote_module_non_scriptable.py
2022-05-18T03:54:16.7659638Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfs0e24g2
2022-05-18T03:54:16.7660561Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfs0e24g2/_remote_module_non_scriptable.py
2022-05-18T03:54:16.8731680Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:54:16.9397085Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:54:16.9695641Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:54:17.0123393Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:54:17.2916744Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:54:17.3019370Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:54:17.3020979Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:54:17.3022011Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:54:17.3023667Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:54:17.3025480Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:54:17.3027132Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:54:17.3031890Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:54:17.7073851Z ok (2.026s)
2022-05-18T03:54:17.7074014Z 
2022-05-18T03:54:17.7074518Z ----------------------------------------------------------------------
2022-05-18T03:54:17.7074995Z Ran 1 test in 2.026s
2022-05-18T03:54:17.7075161Z 
2022-05-18T03:54:17.7075225Z OK
2022-05-18T03:54:17.7075320Z 
2022-05-18T03:54:17.7075401Z Generating XML reports...
2022-05-18T03:54:17.7109597Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220518035415.xml
2022-05-18T03:54:18.4735487Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpiwfozhhm
2022-05-18T03:54:18.4736534Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpiwfozhhm/_remote_module_non_scriptable.py
2022-05-18T03:54:18.7243852Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T03:54:18.7253408Z 
2022-05-18T03:54:18.7253746Z Running tests...
2022-05-18T03:54:18.7254144Z ----------------------------------------------------------------------
2022-05-18T03:54:19.0383540Z   test_graph_for_python_remote_call (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19228
2022-05-18T03:54:19.0405678Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19229
2022-05-18T03:54:19.0429534Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19230
2022-05-18T03:54:19.0453206Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 19231
2022-05-18T03:54:19.7060642Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp76qauv6i
2022-05-18T03:54:19.7061658Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp76qauv6i/_remote_module_non_scriptable.py
2022-05-18T03:54:19.7442676Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptz75dbpf
2022-05-18T03:54:19.7443465Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptz75dbpf/_remote_module_non_scriptable.py
2022-05-18T03:54:19.7771201Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxbrdu336
2022-05-18T03:54:19.7771943Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxbrdu336/_remote_module_non_scriptable.py
2022-05-18T03:54:19.7851161Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnpi7ehsg
2022-05-18T03:54:19.7852062Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnpi7ehsg/_remote_module_non_scriptable.py
2022-05-18T03:54:19.9525139Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:54:19.9918219Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:54:20.0245171Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:54:20.0339369Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:54:20.2672298Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:54:20.2770665Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:54:20.2872387Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:54:20.2873774Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:54:20.2876223Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:54:20.2877419Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:54:20.2878537Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:54:20.2881634Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:54:20.7498730Z ok (2.024s)
2022-05-18T03:54:20.7498909Z 
2022-05-18T03:54:20.7499384Z ----------------------------------------------------------------------
2022-05-18T03:54:20.7499641Z Ran 1 test in 2.024s
2022-05-18T03:54:20.7499758Z 
2022-05-18T03:54:20.7499822Z OK
2022-05-18T03:54:20.7499902Z 
2022-05-18T03:54:20.7500023Z Generating XML reports...
2022-05-18T03:54:20.7532889Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220518035418.xml
2022-05-18T03:54:21.5146569Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpasgepyp4
2022-05-18T03:54:21.5147814Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpasgepyp4/_remote_module_non_scriptable.py
2022-05-18T03:54:21.7665564Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T03:54:21.7675452Z 
2022-05-18T03:54:21.7675576Z Running tests...
2022-05-18T03:54:21.7675961Z ----------------------------------------------------------------------
2022-05-18T03:54:22.0802017Z   test_mixed_requires_grad (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19459
2022-05-18T03:54:22.0824224Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19460
2022-05-18T03:54:22.0847185Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19461
2022-05-18T03:54:22.0871105Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 19462
2022-05-18T03:54:22.7064051Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5fdqoh69
2022-05-18T03:54:22.7065264Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5fdqoh69/_remote_module_non_scriptable.py
2022-05-18T03:54:22.7129751Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3eyk276g
2022-05-18T03:54:22.7130709Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3eyk276g/_remote_module_non_scriptable.py
2022-05-18T03:54:22.7432909Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0g1jkqa4
2022-05-18T03:54:22.7434155Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0g1jkqa4/_remote_module_non_scriptable.py
2022-05-18T03:54:22.7465083Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcyxvvwsi
2022-05-18T03:54:22.7467275Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcyxvvwsi/_remote_module_non_scriptable.py
2022-05-18T03:54:22.9549711Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:54:22.9588844Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:54:22.9902457Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:54:22.9919611Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:54:23.5913252Z ok (1.823s)
2022-05-18T03:54:23.5914496Z 
2022-05-18T03:54:23.5914811Z ----------------------------------------------------------------------
2022-05-18T03:54:23.5915065Z Ran 1 test in 1.824s
2022-05-18T03:54:23.5915182Z 
2022-05-18T03:54:23.5915245Z OK
2022-05-18T03:54:23.5915338Z 
2022-05-18T03:54:23.5915419Z Generating XML reports...
2022-05-18T03:54:23.5948498Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220518035421.xml
2022-05-18T03:54:24.3563335Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp603btb8t
2022-05-18T03:54:24.3564167Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp603btb8t/_remote_module_non_scriptable.py
2022-05-18T03:54:24.6090002Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T03:54:24.6100278Z 
2022-05-18T03:54:24.6100705Z Running tests...
2022-05-18T03:54:24.6101114Z ----------------------------------------------------------------------
2022-05-18T03:54:24.9222801Z   test_multiple_backward (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19698
2022-05-18T03:54:24.9246853Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19699
2022-05-18T03:54:24.9270093Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19700
2022-05-18T03:54:24.9294093Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 19701
2022-05-18T03:54:25.5135979Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqbp5fudf
2022-05-18T03:54:25.5136692Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqbp5fudf/_remote_module_non_scriptable.py
2022-05-18T03:54:25.5650432Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp81bqps06
2022-05-18T03:54:25.5651208Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp81bqps06/_remote_module_non_scriptable.py
2022-05-18T03:54:25.5723705Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmen7u2kr
2022-05-18T03:54:25.5724574Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmen7u2kr/_remote_module_non_scriptable.py
2022-05-18T03:54:25.5782818Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpiyn34h6o
2022-05-18T03:54:25.5784616Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpiyn34h6o/_remote_module_non_scriptable.py
2022-05-18T03:54:25.7606223Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:54:25.8145021Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:54:25.8197994Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:54:25.8282233Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:54:27.1349796Z ok (2.525s)
2022-05-18T03:54:27.1350043Z 
2022-05-18T03:54:27.1350589Z ----------------------------------------------------------------------
2022-05-18T03:54:27.1350862Z Ran 1 test in 2.525s
2022-05-18T03:54:27.1350977Z 
2022-05-18T03:54:27.1351041Z OK
2022-05-18T03:54:27.1351134Z 
2022-05-18T03:54:27.1351229Z Generating XML reports...
2022-05-18T03:54:27.1386821Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220518035424.xml
2022-05-18T03:54:27.9192056Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbus2asdv
2022-05-18T03:54:27.9192947Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbus2asdv/_remote_module_non_scriptable.py
2022-05-18T03:54:28.1728372Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T03:54:28.1738069Z 
2022-05-18T03:54:28.1738508Z Running tests...
2022-05-18T03:54:28.1738918Z ----------------------------------------------------------------------
2022-05-18T03:54:28.4935016Z   test_multiple_backward_with_errors (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19937
2022-05-18T03:54:28.4957834Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19938
2022-05-18T03:54:28.4980811Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19939
2022-05-18T03:54:28.5005593Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 19940
2022-05-18T03:54:29.1464189Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpua9olrr_
2022-05-18T03:54:29.1465293Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpua9olrr_/_remote_module_non_scriptable.py
2022-05-18T03:54:29.1883960Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpipurrv1m
2022-05-18T03:54:29.1885083Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpipurrv1m/_remote_module_non_scriptable.py
2022-05-18T03:54:29.2268125Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvzkrdm1n
2022-05-18T03:54:29.2268993Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvzkrdm1n/_remote_module_non_scriptable.py
2022-05-18T03:54:29.2281742Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz7kl_6bw
2022-05-18T03:54:29.2284355Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz7kl_6bw/_remote_module_non_scriptable.py
2022-05-18T03:54:29.3985775Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:54:29.4383216Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:54:29.4735222Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:54:29.4747874Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:54:29.7392136Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:54:29.7593307Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:54:29.7694892Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:54:29.7696031Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:54:29.7697387Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:54:29.7700442Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:54:29.7701593Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:54:29.7702722Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:54:30.0246981Z [W tensorpipe_agent.cpp:627] RPC agent for worker1 won't send response to request #151 to worker2, as the agent is shutting down
2022-05-18T03:54:30.0261993Z [W tensorpipe_agent.cpp:942] RPC agent for worker2 encountered error when reading incoming response from worker1: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2022-05-18T03:54:30.0263134Z [W tensorpipe_agent.cpp:728] RPC agent for worker0 encountered error when reading incoming request from worker1: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2022-05-18T03:54:30.0264022Z [W tensorpipe_agent.cpp:728] RPC agent for worker2 encountered error when reading incoming request from worker1: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2022-05-18T03:54:30.0264897Z [W tensorpipe_agent.cpp:942] RPC agent for worker0 encountered error when reading incoming response from worker1: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2022-05-18T03:54:30.0265706Z [E container.cpp:257] Could not release Dist Autograd Context on node 1: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2022-05-18T03:54:30.0303815Z [W tensorpipe_agent.cpp:627] RPC agent for worker0 won't send response to request #151 to worker3, as the agent is shutting down
2022-05-18T03:54:30.0318400Z [W tensorpipe_agent.cpp:728] RPC agent for worker2 encountered error when reading incoming request from worker3: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2022-05-18T03:54:30.0321460Z [W tensorpipe_agent.cpp:728] RPC agent for worker0 encountered error when reading incoming request from worker3: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2022-05-18T03:54:30.0326951Z [W tensorpipe_agent.cpp:627] RPC agent for worker1 won't send response to request #152 to worker0, as the agent is shutting down
2022-05-18T03:54:30.0332936Z [W tensorpipe_agent.cpp:728] RPC agent for worker1 encountered error when reading incoming request from worker2: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2022-05-18T03:54:30.0341317Z [W tensorpipe_agent.cpp:728] RPC agent for worker1 encountered error when reading incoming request from worker0: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2022-05-18T03:54:30.0347462Z [W tensorpipe_agent.cpp:728] RPC agent for worker2 encountered error when reading incoming request from worker0: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2022-05-18T03:54:30.0420430Z [W tensorpipe_agent.cpp:728] RPC agent for worker3 encountered error when reading incoming request from worker2: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2022-05-18T03:54:30.0421215Z [W tensorpipe_agent.cpp:728] RPC agent for worker3 encountered error when reading incoming request from worker0: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2022-05-18T03:54:30.0421912Z [E container.cpp:257] Could not release Dist Autograd Context on node 0: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2022-05-18T03:54:30.0493110Z [W tensorpipe_agent.cpp:728] RPC agent for worker0 encountered error when reading incoming request from worker2: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2022-05-18T03:54:30.3052165Z ok (2.131s)
2022-05-18T03:54:30.3052349Z 
2022-05-18T03:54:30.3052773Z ----------------------------------------------------------------------
2022-05-18T03:54:30.3053044Z Ran 1 test in 2.131s
2022-05-18T03:54:30.3053162Z 
2022-05-18T03:54:30.3053225Z OK
2022-05-18T03:54:30.3053302Z 
2022-05-18T03:54:30.3053395Z Generating XML reports...
2022-05-18T03:54:30.3086757Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220518035428.xml
2022-05-18T03:54:31.0645000Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkoo2fw3g
2022-05-18T03:54:31.0645565Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkoo2fw3g/_remote_module_non_scriptable.py
2022-05-18T03:54:31.3161075Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T03:54:31.3170983Z 
2022-05-18T03:54:31.3171119Z Running tests...
2022-05-18T03:54:31.3171713Z ----------------------------------------------------------------------
2022-05-18T03:54:31.6285494Z   test_nested_backward_accumulate_grads (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20188
2022-05-18T03:54:31.6308452Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20189
2022-05-18T03:54:31.6331668Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20190
2022-05-18T03:54:31.6357127Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 20191
2022-05-18T03:54:32.2315083Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp93_csak5
2022-05-18T03:54:32.2316285Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp93_csak5/_remote_module_non_scriptable.py
2022-05-18T03:54:32.2428799Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx1iznl8m
2022-05-18T03:54:32.2429953Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx1iznl8m/_remote_module_non_scriptable.py
2022-05-18T03:54:32.2583830Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi35l923f
2022-05-18T03:54:32.2585052Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi35l923f/_remote_module_non_scriptable.py
2022-05-18T03:54:32.2636241Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcsqlz78o
2022-05-18T03:54:32.2637828Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcsqlz78o/_remote_module_non_scriptable.py
2022-05-18T03:54:32.4804403Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:54:32.4894917Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:54:32.5083183Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:54:32.5140088Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:54:33.0397157Z ok (1.722s)
2022-05-18T03:54:33.0397440Z 
2022-05-18T03:54:33.0397962Z ----------------------------------------------------------------------
2022-05-18T03:54:33.0398259Z Ran 1 test in 1.723s
2022-05-18T03:54:33.0398375Z 
2022-05-18T03:54:33.0398423Z OK
2022-05-18T03:54:33.0398515Z 
2022-05-18T03:54:33.0398610Z Generating XML reports...
2022-05-18T03:54:33.0432415Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220518035431.xml
2022-05-18T03:54:33.8108226Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb0e6z1v6
2022-05-18T03:54:33.8108991Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb0e6z1v6/_remote_module_non_scriptable.py
2022-05-18T03:54:34.0629232Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T03:54:34.0639329Z 
2022-05-18T03:54:34.0639818Z Running tests...
2022-05-18T03:54:34.0640237Z ----------------------------------------------------------------------
2022-05-18T03:54:34.3769100Z   test_nested_context (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20427
2022-05-18T03:54:34.3792270Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20428
2022-05-18T03:54:34.3815173Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20429
2022-05-18T03:54:34.3839890Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 20430
2022-05-18T03:54:35.0502294Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7lfqmq4_
2022-05-18T03:54:35.0503830Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7lfqmq4_/_remote_module_non_scriptable.py
2022-05-18T03:54:35.0882194Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3upu4aet
2022-05-18T03:54:35.0884766Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3upu4aet/_remote_module_non_scriptable.py
2022-05-18T03:54:35.0937114Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5uqlfoj0
2022-05-18T03:54:35.0938726Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5uqlfoj0/_remote_module_non_scriptable.py
2022-05-18T03:54:35.1165075Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0am99cnp
2022-05-18T03:54:35.1166237Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0am99cnp/_remote_module_non_scriptable.py
2022-05-18T03:54:35.2993633Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:54:35.3337589Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:54:35.3436318Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:54:35.3636729Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:54:35.7879471Z ok (1.724s)
2022-05-18T03:54:35.7879979Z 
2022-05-18T03:54:35.7880442Z ----------------------------------------------------------------------
2022-05-18T03:54:35.7880855Z Ran 1 test in 1.724s
2022-05-18T03:54:35.7881036Z 
2022-05-18T03:54:35.7881137Z OK
2022-05-18T03:54:35.7881288Z 
2022-05-18T03:54:35.7881432Z Generating XML reports...
2022-05-18T03:54:35.7915941Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220518035434.xml
2022-05-18T03:54:36.5541562Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkmha3gah
2022-05-18T03:54:36.5542599Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkmha3gah/_remote_module_non_scriptable.py
2022-05-18T03:54:36.8062058Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T03:54:36.8070885Z 
2022-05-18T03:54:36.8071025Z Running tests...
2022-05-18T03:54:36.8071652Z ----------------------------------------------------------------------
2022-05-18T03:54:36.8092625Z   test_no_grad_copy (__main__.TensorPipeDistAutogradTest)
2022-05-18T03:54:37.1211161Z Similar to test in test_autograd.py. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20646
2022-05-18T03:54:37.1234804Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20647
2022-05-18T03:54:37.1257562Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20648
2022-05-18T03:54:37.1282670Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 20649
2022-05-18T03:54:37.7888096Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvv2in2x5
2022-05-18T03:54:37.7888953Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvv2in2x5/_remote_module_non_scriptable.py
2022-05-18T03:54:37.8456802Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmbm8n_e6
2022-05-18T03:54:37.8457982Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmbm8n_e6/_remote_module_non_scriptable.py
2022-05-18T03:54:37.8535599Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyjscb4ao
2022-05-18T03:54:37.8536739Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyjscb4ao/_remote_module_non_scriptable.py
2022-05-18T03:54:37.8603270Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppbv0_3ue
2022-05-18T03:54:37.8605314Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppbv0_3ue/_remote_module_non_scriptable.py
2022-05-18T03:54:38.0368431Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:54:38.0946507Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:54:38.1015059Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:54:38.1065009Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:54:38.5322709Z ok (1.725s)
2022-05-18T03:54:38.5322962Z 
2022-05-18T03:54:38.5323502Z ----------------------------------------------------------------------
2022-05-18T03:54:38.5323835Z Ran 1 test in 1.725s
2022-05-18T03:54:38.5323954Z 
2022-05-18T03:54:38.5324020Z OK
2022-05-18T03:54:38.5324118Z 
2022-05-18T03:54:38.5324199Z Generating XML reports...
2022-05-18T03:54:38.5357632Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220518035436.xml
2022-05-18T03:54:39.2936265Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1h4dyxyg
2022-05-18T03:54:39.2937453Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1h4dyxyg/_remote_module_non_scriptable.py
2022-05-18T03:54:39.5464741Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T03:54:39.5474799Z 
2022-05-18T03:54:39.5474896Z Running tests...
2022-05-18T03:54:39.5476113Z ----------------------------------------------------------------------
2022-05-18T03:54:39.8618787Z   test_no_grad_copy_sparse (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20885
2022-05-18T03:54:39.8641690Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20886
2022-05-18T03:54:39.8664783Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20887
2022-05-18T03:54:39.8688405Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 20888
2022-05-18T03:54:40.5646611Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgcevyp50
2022-05-18T03:54:40.5647385Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgcevyp50/_remote_module_non_scriptable.py
2022-05-18T03:54:40.5689585Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdd_l6wdx
2022-05-18T03:54:40.5691441Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdd_l6wdx/_remote_module_non_scriptable.py
2022-05-18T03:54:40.6048576Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbbaf0kl5
2022-05-18T03:54:40.6049792Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbbaf0kl5/_remote_module_non_scriptable.py
2022-05-18T03:54:40.6155663Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8gsoq3if
2022-05-18T03:54:40.6157125Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8gsoq3if/_remote_module_non_scriptable.py
2022-05-18T03:54:40.8126288Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:54:40.8155337Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:54:40.8520660Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:54:40.8636976Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:54:41.0768812Z /opt/conda/lib/python3.7/site-packages/torch/nn/functional.py:2263: UserWarning: Argument order of nn.functional.embedding_bag was changed. Usage `embedding_bag(weight, input, ...)` is deprecated, and should now be `embedding_bag(input, weight, ...)`.
2022-05-18T03:54:41.0769979Z   "Argument order of nn.functional.embedding_bag was changed. "
2022-05-18T03:54:41.0966855Z /opt/conda/lib/python3.7/site-packages/torch/nn/functional.py:2263: UserWarning: Argument order of nn.functional.embedding_bag was changed. Usage `embedding_bag(weight, input, ...)` is deprecated, and should now be `embedding_bag(input, weight, ...)`.
2022-05-18T03:54:41.0968304Z   "Argument order of nn.functional.embedding_bag was changed. "
2022-05-18T03:54:41.0972497Z /opt/conda/lib/python3.7/site-packages/torch/nn/functional.py:2263: UserWarning: Argument order of nn.functional.embedding_bag was changed. Usage `embedding_bag(weight, input, ...)` is deprecated, and should now be `embedding_bag(input, weight, ...)`.
2022-05-18T03:54:41.0976160Z   "Argument order of nn.functional.embedding_bag was changed. "
2022-05-18T03:54:41.0980263Z /opt/conda/lib/python3.7/site-packages/torch/nn/functional.py:2263: UserWarning: Argument order of nn.functional.embedding_bag was changed. Usage `embedding_bag(weight, input, ...)` is deprecated, and should now be `embedding_bag(input, weight, ...)`.
2022-05-18T03:54:41.0984038Z   "Argument order of nn.functional.embedding_bag was changed. "
2022-05-18T03:54:41.3729929Z ok (1.825s)
2022-05-18T03:54:41.3730203Z 
2022-05-18T03:54:41.3730592Z ----------------------------------------------------------------------
2022-05-18T03:54:41.3730834Z Ran 1 test in 1.825s
2022-05-18T03:54:41.3730961Z 
2022-05-18T03:54:41.3731529Z OK
2022-05-18T03:54:41.3732007Z 
2022-05-18T03:54:41.3732188Z Generating XML reports...
2022-05-18T03:54:41.3765751Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220518035439.xml
2022-05-18T03:54:42.1478406Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8ld5a7va
2022-05-18T03:54:42.1479127Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8ld5a7va/_remote_module_non_scriptable.py
2022-05-18T03:54:42.4024318Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T03:54:42.4033651Z 
2022-05-18T03:54:42.4033785Z Running tests...
2022-05-18T03:54:42.4034397Z ----------------------------------------------------------------------
2022-05-18T03:54:42.7227140Z   test_no_graph_with_tensors_not_require_grad (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21136
2022-05-18T03:54:42.7250389Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21137
2022-05-18T03:54:42.7273600Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21138
2022-05-18T03:54:42.7297912Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 21139
2022-05-18T03:54:43.3400683Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc9x5i7zc
2022-05-18T03:54:43.3401480Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc9x5i7zc/_remote_module_non_scriptable.py
2022-05-18T03:54:43.3505006Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_u2m6kpu
2022-05-18T03:54:43.3505749Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_u2m6kpu/_remote_module_non_scriptable.py
2022-05-18T03:54:43.3718108Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw8vq2dh1
2022-05-18T03:54:43.3718841Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw8vq2dh1/_remote_module_non_scriptable.py
2022-05-18T03:54:43.3921953Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn47ym99n
2022-05-18T03:54:43.3922939Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn47ym99n/_remote_module_non_scriptable.py
2022-05-18T03:54:43.5909201Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:54:43.5986117Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:54:43.6204386Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:54:43.6394105Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:54:43.9131105Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:54:43.9233611Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:54:43.9234533Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:54:43.9235328Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:54:43.9236575Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:54:43.9237742Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:54:43.9238865Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:54:43.9333760Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:54:44.3340628Z ok (1.930s)
2022-05-18T03:54:44.3341068Z 
2022-05-18T03:54:44.3341965Z ----------------------------------------------------------------------
2022-05-18T03:54:44.3342712Z Ran 1 test in 1.931s
2022-05-18T03:54:44.3342865Z 
2022-05-18T03:54:44.3343082Z OK
2022-05-18T03:54:44.3343218Z 
2022-05-18T03:54:44.3343355Z Generating XML reports...
2022-05-18T03:54:44.3378425Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220518035442.xml
2022-05-18T03:54:45.1040986Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdn_rfax2
2022-05-18T03:54:45.1041447Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdn_rfax2/_remote_module_non_scriptable.py
2022-05-18T03:54:45.3562033Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T03:54:45.3571458Z 
2022-05-18T03:54:45.3571591Z Running tests...
2022-05-18T03:54:45.3572028Z ----------------------------------------------------------------------
2022-05-18T03:54:45.6751116Z   test_no_graph_with_tensors_not_require_grad_remote (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21367
2022-05-18T03:54:45.6774022Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21368
2022-05-18T03:54:45.6797611Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21369
2022-05-18T03:54:45.6821279Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 21370
2022-05-18T03:54:46.2487957Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt5cdebvt
2022-05-18T03:54:46.2488805Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt5cdebvt/_remote_module_non_scriptable.py
2022-05-18T03:54:46.2970062Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp56h3wt8z
2022-05-18T03:54:46.2970724Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7wc0m03h
2022-05-18T03:54:46.2971322Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp56h3wt8z/_remote_module_non_scriptable.py
2022-05-18T03:54:46.2972152Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7wc0m03h/_remote_module_non_scriptable.py
2022-05-18T03:54:46.2981742Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptabim0go
2022-05-18T03:54:46.2983716Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptabim0go/_remote_module_non_scriptable.py
2022-05-18T03:54:46.4958360Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:54:46.5446924Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:54:46.5456199Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:54:46.5458874Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:54:46.8051110Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:54:46.8152055Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:54:46.8253091Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:54:46.8254656Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:54:46.8255753Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:54:46.8257298Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:54:46.8258171Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:54:46.8259348Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:54:47.1863045Z ok (1.829s)
2022-05-18T03:54:47.1863275Z 
2022-05-18T03:54:47.1863816Z ----------------------------------------------------------------------
2022-05-18T03:54:47.1864160Z Ran 1 test in 1.829s
2022-05-18T03:54:47.1864504Z 
2022-05-18T03:54:47.1864553Z OK
2022-05-18T03:54:47.1864647Z 
2022-05-18T03:54:47.1864741Z Generating XML reports...
2022-05-18T03:54:47.1897189Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220518035445.xml
2022-05-18T03:54:47.9616268Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0qcd_469
2022-05-18T03:54:47.9617502Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0qcd_469/_remote_module_non_scriptable.py
2022-05-18T03:54:48.2163928Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T03:54:48.2173759Z 
2022-05-18T03:54:48.2173940Z Running tests...
2022-05-18T03:54:48.2174343Z ----------------------------------------------------------------------
2022-05-18T03:54:48.5355039Z   test_post_hooks (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21598
2022-05-18T03:54:48.5377533Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21599
2022-05-18T03:54:48.5400796Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21600
2022-05-18T03:54:48.5425146Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 21601
2022-05-18T03:54:49.1757497Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp46i0enyg
2022-05-18T03:54:49.1758463Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp46i0enyg/_remote_module_non_scriptable.py
2022-05-18T03:54:49.1845762Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj2kksolx
2022-05-18T03:54:49.1848231Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj2kksolx/_remote_module_non_scriptable.py
2022-05-18T03:54:49.2196669Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0w1aey44
2022-05-18T03:54:49.2197876Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0w1aey44/_remote_module_non_scriptable.py
2022-05-18T03:54:49.2231017Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4phbx1ur
2022-05-18T03:54:49.2232357Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4phbx1ur/_remote_module_non_scriptable.py
2022-05-18T03:54:49.4266882Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:54:49.4292703Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:54:49.4701897Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:54:49.4715684Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:54:49.9465549Z ok (1.729s)
2022-05-18T03:54:49.9465813Z 
2022-05-18T03:54:49.9466209Z ----------------------------------------------------------------------
2022-05-18T03:54:49.9466464Z Ran 1 test in 1.729s
2022-05-18T03:54:49.9466605Z 
2022-05-18T03:54:49.9466653Z OK
2022-05-18T03:54:49.9466748Z 
2022-05-18T03:54:49.9466844Z Generating XML reports...
2022-05-18T03:54:49.9501856Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220518035448.xml
2022-05-18T03:54:50.7273503Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppv_xtw3r
2022-05-18T03:54:50.7274286Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppv_xtw3r/_remote_module_non_scriptable.py
2022-05-18T03:54:50.9784557Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T03:54:50.9793458Z 
2022-05-18T03:54:50.9793782Z Running tests...
2022-05-18T03:54:50.9794495Z ----------------------------------------------------------------------
2022-05-18T03:54:51.2930951Z   test_remote_complex_args (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21837
2022-05-18T03:54:51.2953805Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21838
2022-05-18T03:54:51.2977032Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21839
2022-05-18T03:54:51.3001134Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 21840
2022-05-18T03:54:51.8972310Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2jcl8idw
2022-05-18T03:54:51.8973136Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2jcl8idw/_remote_module_non_scriptable.py
2022-05-18T03:54:51.9256067Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpiwynja42
2022-05-18T03:54:51.9257002Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpiwynja42/_remote_module_non_scriptable.py
2022-05-18T03:54:51.9360573Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnxv3avk2
2022-05-18T03:54:51.9361589Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnxv3avk2/_remote_module_non_scriptable.py
2022-05-18T03:54:51.9368568Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw61f6092
2022-05-18T03:54:51.9370899Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw61f6092/_remote_module_non_scriptable.py
2022-05-18T03:54:52.1438438Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:54:52.1730484Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:54:52.1831971Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:54:52.1856388Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:54:52.7041730Z ok (1.724s)
2022-05-18T03:54:52.7041895Z 
2022-05-18T03:54:52.7042212Z ----------------------------------------------------------------------
2022-05-18T03:54:52.7042467Z Ran 1 test in 1.725s
2022-05-18T03:54:52.7042599Z 
2022-05-18T03:54:52.7042649Z OK
2022-05-18T03:54:52.7042741Z 
2022-05-18T03:54:52.7042835Z Generating XML reports...
2022-05-18T03:54:52.7077737Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220518035450.xml
2022-05-18T03:54:53.4723501Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqd5i000h
2022-05-18T03:54:53.4724311Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqd5i000h/_remote_module_non_scriptable.py
2022-05-18T03:54:53.7244490Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T03:54:53.7253661Z 
2022-05-18T03:54:53.7253986Z Running tests...
2022-05-18T03:54:53.7254381Z ----------------------------------------------------------------------
2022-05-18T03:54:54.0405203Z   test_rpc_complex_args (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22056
2022-05-18T03:54:54.0427213Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22057
2022-05-18T03:54:54.0449843Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22058
2022-05-18T03:54:54.0474206Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 22059
2022-05-18T03:54:54.6591066Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3fqupizo
2022-05-18T03:54:54.6657204Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3fqupizo/_remote_module_non_scriptable.py
2022-05-18T03:54:54.6903338Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdgsbw0ul
2022-05-18T03:54:54.6905002Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdgsbw0ul/_remote_module_non_scriptable.py
2022-05-18T03:54:54.6977311Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp50ogshse
2022-05-18T03:54:54.6978878Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp50ogshse/_remote_module_non_scriptable.py
2022-05-18T03:54:54.7137928Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd6_dedu7
2022-05-18T03:54:54.7139552Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd6_dedu7/_remote_module_non_scriptable.py
2022-05-18T03:54:54.9084027Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:54:54.9401450Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:54:54.9479779Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:54:54.9635427Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:54:55.4514169Z ok (1.726s)
2022-05-18T03:54:55.4514391Z 
2022-05-18T03:54:55.4514847Z ----------------------------------------------------------------------
2022-05-18T03:54:55.4515185Z Ran 1 test in 1.726s
2022-05-18T03:54:55.4515291Z 
2022-05-18T03:54:55.4515359Z OK
2022-05-18T03:54:55.4515452Z 
2022-05-18T03:54:55.4515546Z Generating XML reports...
2022-05-18T03:54:55.4552676Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220518035453.xml
2022-05-18T03:54:56.2394297Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppfv0w2wb
2022-05-18T03:54:56.2395434Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppfv0w2wb/_remote_module_non_scriptable.py
2022-05-18T03:54:56.4940364Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T03:54:56.4950440Z 
2022-05-18T03:54:56.4950565Z Running tests...
2022-05-18T03:54:56.4951154Z ----------------------------------------------------------------------
2022-05-18T03:54:56.8124321Z   test_thread_local_context_id (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22275
2022-05-18T03:54:56.8147136Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22276
2022-05-18T03:54:56.8170090Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22277
2022-05-18T03:54:56.8195034Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 22278
2022-05-18T03:54:57.3978481Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp55bqf66w
2022-05-18T03:54:57.3979584Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp55bqf66w/_remote_module_non_scriptable.py
2022-05-18T03:54:57.4208646Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpetvbzsgf
2022-05-18T03:54:57.4209561Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpetvbzsgf/_remote_module_non_scriptable.py
2022-05-18T03:54:57.4457699Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3unke4ck
2022-05-18T03:54:57.4458524Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3unke4ck/_remote_module_non_scriptable.py
2022-05-18T03:54:57.4593551Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgzgbgmm1
2022-05-18T03:54:57.4594979Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgzgbgmm1/_remote_module_non_scriptable.py
2022-05-18T03:54:57.6471656Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:54:57.6701727Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:54:57.6944432Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:54:57.7084615Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:54:59.2249740Z ok (2.730s)
2022-05-18T03:54:59.2249976Z 
2022-05-18T03:54:59.2250416Z ----------------------------------------------------------------------
2022-05-18T03:54:59.2251105Z Ran 1 test in 2.730s
2022-05-18T03:54:59.2251287Z 
2022-05-18T03:54:59.2251387Z OK
2022-05-18T03:54:59.2251534Z 
2022-05-18T03:54:59.2251663Z Generating XML reports...
2022-05-18T03:54:59.2285747Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220518035456.xml
2022-05-18T03:54:59.9899548Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd8_gewa_
2022-05-18T03:54:59.9900446Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd8_gewa_/_remote_module_non_scriptable.py
2022-05-18T03:55:00.2441466Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T03:55:00.2450910Z 
2022-05-18T03:55:00.2450988Z Running tests...
2022-05-18T03:55:00.2451569Z ----------------------------------------------------------------------
2022-05-18T03:55:00.5571267Z   test_trainer_ps (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22514
2022-05-18T03:55:00.5593284Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22515
2022-05-18T03:55:00.5616425Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22516
2022-05-18T03:55:00.5640992Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 22517
2022-05-18T03:55:01.1483313Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp47m4qqw1
2022-05-18T03:55:01.1484319Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp47m4qqw1/_remote_module_non_scriptable.py
2022-05-18T03:55:01.1775757Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu3826hd0
2022-05-18T03:55:01.1777054Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu3826hd0/_remote_module_non_scriptable.py
2022-05-18T03:55:01.1867998Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzxz9d378
2022-05-18T03:55:01.1869216Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzxz9d378/_remote_module_non_scriptable.py
2022-05-18T03:55:01.2049560Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqmuup4iv
2022-05-18T03:55:01.2050439Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqmuup4iv/_remote_module_non_scriptable.py
2022-05-18T03:55:01.3998252Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:55:01.4265338Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:55:01.4363340Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:55:01.4511270Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:55:02.0685388Z ok (1.823s)
2022-05-18T03:55:02.0685644Z 
2022-05-18T03:55:02.0686165Z ----------------------------------------------------------------------
2022-05-18T03:55:02.0686531Z Ran 1 test in 1.823s
2022-05-18T03:55:02.0686651Z 
2022-05-18T03:55:02.0686716Z OK
2022-05-18T03:55:02.0686811Z 
2022-05-18T03:55:02.0686906Z Generating XML reports...
2022-05-18T03:55:02.0720100Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220518035500.xml
2022-05-18T03:55:02.8409137Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxjxkpjus
2022-05-18T03:55:02.8410188Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxjxkpjus/_remote_module_non_scriptable.py
2022-05-18T03:55:03.0933719Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T03:55:03.0943619Z 
2022-05-18T03:55:03.0944070Z Running tests...
2022-05-18T03:55:03.0944677Z ----------------------------------------------------------------------
2022-05-18T03:55:03.4115418Z   test_trainer_ps_torchscript_functions (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22753
2022-05-18T03:55:03.4136975Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22754
2022-05-18T03:55:03.4160409Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22755
2022-05-18T03:55:03.4185041Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 22756
2022-05-18T03:55:04.0788537Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprz5mq7c9
2022-05-18T03:55:04.0789488Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprz5mq7c9/_remote_module_non_scriptable.py
2022-05-18T03:55:04.1628444Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmch9mckx
2022-05-18T03:55:04.1628985Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmch9mckx/_remote_module_non_scriptable.py
2022-05-18T03:55:04.2042932Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpauuaaetk
2022-05-18T03:55:04.2043716Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpauuaaetk/_remote_module_non_scriptable.py
2022-05-18T03:55:04.2194768Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvsmtghfj
2022-05-18T03:55:04.2195508Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvsmtghfj/_remote_module_non_scriptable.py
2022-05-18T03:55:04.3278996Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:55:04.4129130Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:55:04.4514675Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:55:04.4676857Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:55:05.1230418Z ok (2.028s)
2022-05-18T03:55:05.1230684Z 
2022-05-18T03:55:05.1231192Z ----------------------------------------------------------------------
2022-05-18T03:55:05.1231440Z Ran 1 test in 2.029s
2022-05-18T03:55:05.1231568Z 
2022-05-18T03:55:05.1231631Z OK
2022-05-18T03:55:05.1231724Z 
2022-05-18T03:55:05.1231823Z Generating XML reports...
2022-05-18T03:55:05.1264723Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220518035503.xml
2022-05-18T03:55:05.9067745Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6l8cqn05
2022-05-18T03:55:05.9068413Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6l8cqn05/_remote_module_non_scriptable.py
2022-05-18T03:55:06.1616050Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T03:55:06.1626033Z 
2022-05-18T03:55:06.1626194Z Running tests...
2022-05-18T03:55:06.1626604Z ----------------------------------------------------------------------
2022-05-18T03:55:06.4778828Z   test_worker_ids_recorded (__main__.TensorPipeDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22992
2022-05-18T03:55:06.4803352Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22993
2022-05-18T03:55:06.4826826Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22994
2022-05-18T03:55:06.4852276Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 22995
2022-05-18T03:55:07.0835998Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbbpu3hf8
2022-05-18T03:55:07.0836783Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbbpu3hf8/_remote_module_non_scriptable.py
2022-05-18T03:55:07.0902522Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjs7evh8u
2022-05-18T03:55:07.0903889Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjs7evh8u/_remote_module_non_scriptable.py
2022-05-18T03:55:07.0993806Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5_u0yq48
2022-05-18T03:55:07.0995209Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5_u0yq48/_remote_module_non_scriptable.py
2022-05-18T03:55:07.1176240Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvusmt0t8
2022-05-18T03:55:07.1177667Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvusmt0t8/_remote_module_non_scriptable.py
2022-05-18T03:55:07.3339514Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:55:07.3402281Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:55:07.3487182Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:55:07.3672817Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:55:07.8891201Z ok (1.726s)
2022-05-18T03:55:07.8891442Z 
2022-05-18T03:55:07.8891976Z ----------------------------------------------------------------------
2022-05-18T03:55:07.8892408Z Ran 1 test in 1.726s
2022-05-18T03:55:07.8892536Z 
2022-05-18T03:55:07.8892585Z OK
2022-05-18T03:55:07.8892675Z 
2022-05-18T03:55:07.8892771Z Generating XML reports...
2022-05-18T03:55:07.8926421Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220518035506.xml
2022-05-18T03:55:08.6610884Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpptz2zcd8
2022-05-18T03:55:08.6611821Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpptz2zcd8/_remote_module_non_scriptable.py
2022-05-18T03:55:08.9137809Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T03:55:08.9147980Z 
2022-05-18T03:55:08.9148326Z Running tests...
2022-05-18T03:55:08.9148727Z ----------------------------------------------------------------------
2022-05-18T03:55:09.2228391Z   test_dist_optim (__main__.TensorPipeDistOptimizerTest) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/72997 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure IN_CI is not set and you are not using the flag --import-disabled-tests. (0.308s)
2022-05-18T03:55:09.2229044Z 
2022-05-18T03:55:09.2229257Z ----------------------------------------------------------------------
2022-05-18T03:55:09.2229565Z Ran 1 test in 0.308s
2022-05-18T03:55:09.2229692Z 
2022-05-18T03:55:09.2229751Z OK (skipped=1)
2022-05-18T03:55:09.2229860Z 
2022-05-18T03:55:09.2229949Z Generating XML reports...
2022-05-18T03:55:09.2251613Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistOptimizerTest-20220518035508.xml
2022-05-18T03:55:09.9469232Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp69gqra3t
2022-05-18T03:55:09.9469993Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp69gqra3t/_remote_module_non_scriptable.py
2022-05-18T03:55:10.2025517Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T03:55:10.2035164Z 
2022-05-18T03:55:10.2035304Z Running tests...
2022-05-18T03:55:10.2035719Z ----------------------------------------------------------------------
2022-05-18T03:55:10.5224472Z   test_dist_optim_exception (__main__.TensorPipeDistOptimizerTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23221
2022-05-18T03:55:10.5246080Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23222
2022-05-18T03:55:10.5269425Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23223
2022-05-18T03:55:10.5293553Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 23224
2022-05-18T03:55:11.1664548Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj71nswtn
2022-05-18T03:55:11.1665469Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp79k3yyun
2022-05-18T03:55:11.1666122Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj71nswtn/_remote_module_non_scriptable.py
2022-05-18T03:55:11.1668807Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp79k3yyun/_remote_module_non_scriptable.py
2022-05-18T03:55:11.1745202Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpng2by88_
2022-05-18T03:55:11.1747165Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpng2by88_/_remote_module_non_scriptable.py
2022-05-18T03:55:11.1975773Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn32j13ke
2022-05-18T03:55:11.1976748Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn32j13ke/_remote_module_non_scriptable.py
2022-05-18T03:55:11.4146124Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:55:11.4156506Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:55:11.4222174Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:55:11.4437461Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:55:11.6532750Z WARNING:torch.distributed.optim.optimizer:Creating the optimizer <class 'torch.testing._internal.distributed.rpc.dist_optimizer_test.FailingOptimizer'> without TorchScript support, this might result in slow computation time in multithreading environment(i.e. Distributed Model Parallel training on CPU) due to the Python's Global Interpreter Lock (GIL). Please file an issue if you need this optimizer in TorchScript. 
2022-05-18T03:55:11.6619541Z On WorkerInfo(id=1, name=worker1):
2022-05-18T03:55:11.6621991Z ValueError('Error running optimizer.')
2022-05-18T03:55:11.6622514Z Traceback (most recent call last):
2022-05-18T03:55:11.6623548Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function
2022-05-18T03:55:11.6624307Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-05-18T03:55:11.6625259Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 95, in _local_optimizer_step
2022-05-18T03:55:11.6625938Z     local_optim.step(autograd_ctx_id)
2022-05-18T03:55:11.6626777Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 85, in step
2022-05-18T03:55:11.6627371Z     self.optim.step()
2022-05-18T03:55:11.6628126Z   File "/opt/conda/lib/python3.7/site-packages/torch/optim/optimizer.py", line 88, in wrapper
2022-05-18T03:55:11.6628705Z     return func(*args, **kwargs)
2022-05-18T03:55:11.6629615Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/dist_optimizer_test.py", line 39, in step
2022-05-18T03:55:11.6630324Z     raise ValueError("Error running optimizer.")
2022-05-18T03:55:11.6630836Z ValueError: Error running optimizer.
2022-05-18T03:55:11.6631311Z On WorkerInfo(id=2, name=worker2):
2022-05-18T03:55:11.6631852Z ValueError('Error running optimizer.')
2022-05-18T03:55:11.6632328Z Traceback (most recent call last):
2022-05-18T03:55:11.6633164Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function
2022-05-18T03:55:11.6633892Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-05-18T03:55:11.6634823Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 95, in _local_optimizer_step
2022-05-18T03:55:11.6635479Z     local_optim.step(autograd_ctx_id)
2022-05-18T03:55:11.6636304Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 85, in step
2022-05-18T03:55:11.6637234Z     self.optim.step()
2022-05-18T03:55:11.6637967Z   File "/opt/conda/lib/python3.7/site-packages/torch/optim/optimizer.py", line 88, in wrapper
2022-05-18T03:55:11.6638646Z     return func(*args, **kwargs)
2022-05-18T03:55:11.6639561Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/dist_optimizer_test.py", line 39, in step
2022-05-18T03:55:11.6640260Z     raise ValueError("Error running optimizer.")
2022-05-18T03:55:11.6640765Z ValueError: Error running optimizer.
2022-05-18T03:55:11.6641053Z 
2022-05-18T03:55:11.6641062Z 
2022-05-18T03:55:11.6729517Z WARNING:torch.distributed.optim.optimizer:Creating the optimizer <class 'torch.testing._internal.distributed.rpc.dist_optimizer_test.FailingOptimizer'> without TorchScript support, this might result in slow computation time in multithreading environment(i.e. Distributed Model Parallel training on CPU) due to the Python's Global Interpreter Lock (GIL). Please file an issue if you need this optimizer in TorchScript. 
2022-05-18T03:55:11.6732098Z WARNING:torch.distributed.optim.optimizer:Creating the optimizer <class 'torch.testing._internal.distributed.rpc.dist_optimizer_test.FailingOptimizer'> without TorchScript support, this might result in slow computation time in multithreading environment(i.e. Distributed Model Parallel training on CPU) due to the Python's Global Interpreter Lock (GIL). Please file an issue if you need this optimizer in TorchScript. 
2022-05-18T03:55:11.6734514Z WARNING:torch.distributed.optim.optimizer:Creating the optimizer <class 'torch.testing._internal.distributed.rpc.dist_optimizer_test.FailingOptimizer'> without TorchScript support, this might result in slow computation time in multithreading environment(i.e. Distributed Model Parallel training on CPU) due to the Python's Global Interpreter Lock (GIL). Please file an issue if you need this optimizer in TorchScript. 
2022-05-18T03:55:11.7203262Z On WorkerInfo(id=1, name=worker1):
2022-05-18T03:55:11.7203889Z ValueError('Error running optimizer.')
2022-05-18T03:55:11.7204241Z Traceback (most recent call last):
2022-05-18T03:55:11.7204989Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function
2022-05-18T03:55:11.7205514Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-05-18T03:55:11.7206186Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 95, in _local_optimizer_step
2022-05-18T03:55:11.7206715Z     local_optim.step(autograd_ctx_id)
2022-05-18T03:55:11.7207234Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 85, in step
2022-05-18T03:55:11.7207525Z     self.optim.step()
2022-05-18T03:55:11.7207863Z   File "/opt/conda/lib/python3.7/site-packages/torch/optim/optimizer.py", line 88, in wrapper
2022-05-18T03:55:11.7208138Z     return func(*args, **kwargs)
2022-05-18T03:55:11.7208547Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/dist_optimizer_test.py", line 39, in step
2022-05-18T03:55:11.7209007Z     raise ValueError("Error running optimizer.")
2022-05-18T03:55:11.7209411Z ValueError: Error running optimizer.
2022-05-18T03:55:11.7209628Z 
2022-05-18T03:55:11.7209814Z On WorkerInfo(id=0, name=worker0):
2022-05-18T03:55:11.7210224Z ValueError('Error running optimizer.')
2022-05-18T03:55:11.7210527Z Traceback (most recent call last):
2022-05-18T03:55:11.7211137Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function
2022-05-18T03:55:11.7211675Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-05-18T03:55:11.7212376Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 95, in _local_optimizer_step
2022-05-18T03:55:11.7212876Z     local_optim.step(autograd_ctx_id)
2022-05-18T03:55:11.7213495Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 85, in step
2022-05-18T03:55:11.7214239Z     self.optim.step()
2022-05-18T03:55:11.7214786Z   File "/opt/conda/lib/python3.7/site-packages/torch/optim/optimizer.py", line 88, in wrapper
2022-05-18T03:55:11.7215207Z     return func(*args, **kwargs)
2022-05-18T03:55:11.7216026Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/dist_optimizer_test.py", line 39, in step
2022-05-18T03:55:11.7216363Z     raise ValueError("Error running optimizer.")
2022-05-18T03:55:11.7216676Z ValueError: Error running optimizer.
2022-05-18T03:55:11.7216811Z 
2022-05-18T03:55:11.7519151Z On WorkerInfo(id=2, name=worker2):
2022-05-18T03:55:11.7519527Z ValueError('Error running optimizer.')
2022-05-18T03:55:11.7519945Z Traceback (most recent call last):
2022-05-18T03:55:11.7520811Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function
2022-05-18T03:55:11.7521438Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-05-18T03:55:11.7522049Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 95, in _local_optimizer_step
2022-05-18T03:55:11.7522539Z     local_optim.step(autograd_ctx_id)
2022-05-18T03:55:11.7523198Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 85, in step
2022-05-18T03:55:11.7523726Z     self.optim.step()
2022-05-18T03:55:11.7524321Z   File "/opt/conda/lib/python3.7/site-packages/torch/optim/optimizer.py", line 88, in wrapper
2022-05-18T03:55:11.7524702Z     return func(*args, **kwargs)
2022-05-18T03:55:11.7525346Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/dist_optimizer_test.py", line 39, in step
2022-05-18T03:55:11.7525831Z     raise ValueError("Error running optimizer.")
2022-05-18T03:55:11.7526176Z ValueError: Error running optimizer.
2022-05-18T03:55:11.7526390Z 
2022-05-18T03:55:11.7526529Z On WorkerInfo(id=0, name=worker0):
2022-05-18T03:55:11.7526957Z ValueError('Error running optimizer.')
2022-05-18T03:55:11.7527314Z Traceback (most recent call last):
2022-05-18T03:55:11.7527957Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function
2022-05-18T03:55:11.7528557Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-05-18T03:55:11.7529320Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 95, in _local_optimizer_step
2022-05-18T03:55:11.7529886Z     local_optim.step(autograd_ctx_id)
2022-05-18T03:55:11.7530577Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 85, in step
2022-05-18T03:55:11.7531085Z     self.optim.step()
2022-05-18T03:55:11.7531875Z   File "/opt/conda/lib/python3.7/site-packages/torch/optim/optimizer.py", line 88, in wrapper
2022-05-18T03:55:11.7533271Z     return func(*args, **kwargs)
2022-05-18T03:55:11.7535741Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/dist_optimizer_test.py", line 39, in step
2022-05-18T03:55:11.7537581Z     raise ValueError("Error running optimizer.")
2022-05-18T03:55:11.7538093Z ValueError: Error running optimizer.
2022-05-18T03:55:11.7538502Z 
2022-05-18T03:55:11.7539490Z On WorkerInfo(id=3, name=worker3):
2022-05-18T03:55:11.7540044Z ValueError('Error running optimizer.')
2022-05-18T03:55:11.7540525Z Traceback (most recent call last):
2022-05-18T03:55:11.7541702Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function
2022-05-18T03:55:11.7542390Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-05-18T03:55:11.7543485Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 95, in _local_optimizer_step
2022-05-18T03:55:11.7544175Z     local_optim.step(autograd_ctx_id)
2022-05-18T03:55:11.7545012Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 85, in step
2022-05-18T03:55:11.7545587Z     self.optim.step()
2022-05-18T03:55:11.7546587Z   File "/opt/conda/lib/python3.7/site-packages/torch/optim/optimizer.py", line 88, in wrapper
2022-05-18T03:55:11.7547172Z     return func(*args, **kwargs)
2022-05-18T03:55:11.7548093Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/dist_optimizer_test.py", line 39, in step
2022-05-18T03:55:11.7554082Z     raise ValueError("Error running optimizer.")
2022-05-18T03:55:11.7554540Z ValueError: Error running optimizer.
2022-05-18T03:55:11.7555127Z 
2022-05-18T03:55:11.7555376Z On WorkerInfo(id=3, name=worker3):
2022-05-18T03:55:11.7555766Z ValueError('Error running optimizer.')
2022-05-18T03:55:11.7557821Z Traceback (most recent call last):
2022-05-18T03:55:11.7558297Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function
2022-05-18T03:55:11.7559431Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-05-18T03:55:11.7560167Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 95, in _local_optimizer_step
2022-05-18T03:55:11.7560643Z     local_optim.step(autograd_ctx_id)
2022-05-18T03:55:11.7561261Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 85, in step
2022-05-18T03:55:11.7561768Z     self.optim.step()
2022-05-18T03:55:11.7562142Z   File "/opt/conda/lib/python3.7/site-packages/torch/optim/optimizer.py", line 88, in wrapper
2022-05-18T03:55:11.7581498Z     return func(*args, **kwargs)
2022-05-18T03:55:11.7582511Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/dist_optimizer_test.py", line 39, in step
2022-05-18T03:55:11.7583385Z     raise ValueError("Error running optimizer.")
2022-05-18T03:55:11.7583874Z ValueError: Error running optimizer.
2022-05-18T03:55:11.7584169Z 
2022-05-18T03:55:12.0335409Z ok (1.830s)
2022-05-18T03:55:12.0335624Z 
2022-05-18T03:55:12.0336080Z ----------------------------------------------------------------------
2022-05-18T03:55:12.0336360Z Ran 1 test in 1.830s
2022-05-18T03:55:12.0336551Z 
2022-05-18T03:55:12.0336614Z OK
2022-05-18T03:55:12.0336709Z 
2022-05-18T03:55:12.0336789Z Generating XML reports...
2022-05-18T03:55:12.0372128Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistOptimizerTest-20220518035510.xml
2022-05-18T03:55:12.8210826Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv9evy382
2022-05-18T03:55:12.8211524Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv9evy382/_remote_module_non_scriptable.py
2022-05-18T03:55:13.0713540Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T03:55:13.0723011Z 
2022-05-18T03:55:13.0723111Z Running tests...
2022-05-18T03:55:13.0723800Z ----------------------------------------------------------------------
2022-05-18T03:55:13.3941302Z   test_dist_optim_exception_on_constructor (__main__.TensorPipeDistOptimizerTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23460
2022-05-18T03:55:13.3964204Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23461
2022-05-18T03:55:13.3987254Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23462
2022-05-18T03:55:13.4011317Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 23463
2022-05-18T03:55:14.0010263Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpevidfwgy
2022-05-18T03:55:14.0011174Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpevidfwgy/_remote_module_non_scriptable.py
2022-05-18T03:55:14.0298587Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4nvk1ag6
2022-05-18T03:55:14.0299384Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4nvk1ag6/_remote_module_non_scriptable.py
2022-05-18T03:55:14.0465772Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb4acok26
2022-05-18T03:55:14.0466991Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb4acok26/_remote_module_non_scriptable.py
2022-05-18T03:55:14.0504720Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe3rvjnhh
2022-05-18T03:55:14.0506014Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe3rvjnhh/_remote_module_non_scriptable.py
2022-05-18T03:55:14.2519865Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:55:14.2793369Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:55:14.2972149Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:55:14.2972806Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:55:14.5051443Z WARNING:torch.distributed.optim.optimizer:Creating the optimizer <class 'torch.testing._internal.distributed.rpc.dist_optimizer_test.OptimizerFailingOnConstructor'> without TorchScript support, this might result in slow computation time in multithreading environment(i.e. Distributed Model Parallel training on CPU) due to the Python's Global Interpreter Lock (GIL). Please file an issue if you need this optimizer in TorchScript. 
2022-05-18T03:55:14.5061355Z On WorkerInfo(id=1, name=worker1):
2022-05-18T03:55:14.5062054Z ValueError('Error creating optimizer.')
2022-05-18T03:55:14.5062547Z Traceback (most recent call last):
2022-05-18T03:55:14.5063578Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function
2022-05-18T03:55:14.5066201Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-05-18T03:55:14.5067452Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 90, in _new_local_optimizer
2022-05-18T03:55:14.5068424Z     _LocalOptimizer(optim_cls, local_params_rref, *args, **kwargs))
2022-05-18T03:55:14.5069588Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 77, in __init__
2022-05-18T03:55:14.5070200Z     **kwargs)
2022-05-18T03:55:14.5071094Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/dist_optimizer_test.py", line 45, in __init__
2022-05-18T03:55:14.5071839Z     raise ValueError("Error creating optimizer.")
2022-05-18T03:55:14.5072345Z ValueError: Error creating optimizer.
2022-05-18T03:55:14.5072634Z 
2022-05-18T03:55:14.5072840Z On WorkerInfo(id=2, name=worker2):
2022-05-18T03:55:14.5075659Z ValueError('Error creating optimizer.')
2022-05-18T03:55:14.5076126Z Traceback (most recent call last):
2022-05-18T03:55:14.5076831Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function
2022-05-18T03:55:14.5077481Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-05-18T03:55:14.5078215Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 90, in _new_local_optimizer
2022-05-18T03:55:14.5078818Z     _LocalOptimizer(optim_cls, local_params_rref, *args, **kwargs))
2022-05-18T03:55:14.5079743Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 77, in __init__
2022-05-18T03:55:14.5080298Z     **kwargs)
2022-05-18T03:55:14.5081178Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/dist_optimizer_test.py", line 45, in __init__
2022-05-18T03:55:14.5081910Z     raise ValueError("Error creating optimizer.")
2022-05-18T03:55:14.5082415Z ValueError: Error creating optimizer.
2022-05-18T03:55:14.5082687Z 
2022-05-18T03:55:14.5253823Z WARNING:torch.distributed.optim.optimizer:Creating the optimizer <class 'torch.testing._internal.distributed.rpc.dist_optimizer_test.OptimizerFailingOnConstructor'> without TorchScript support, this might result in slow computation time in multithreading environment(i.e. Distributed Model Parallel training on CPU) due to the Python's Global Interpreter Lock (GIL). Please file an issue if you need this optimizer in TorchScript. 
2022-05-18T03:55:14.5256776Z WARNING:torch.distributed.optim.optimizer:Creating the optimizer <class 'torch.testing._internal.distributed.rpc.dist_optimizer_test.OptimizerFailingOnConstructor'> without TorchScript support, this might result in slow computation time in multithreading environment(i.e. Distributed Model Parallel training on CPU) due to the Python's Global Interpreter Lock (GIL). Please file an issue if you need this optimizer in TorchScript. 
2022-05-18T03:55:14.5259161Z WARNING:torch.distributed.optim.optimizer:Creating the optimizer <class 'torch.testing._internal.distributed.rpc.dist_optimizer_test.OptimizerFailingOnConstructor'> without TorchScript support, this might result in slow computation time in multithreading environment(i.e. Distributed Model Parallel training on CPU) due to the Python's Global Interpreter Lock (GIL). Please file an issue if you need this optimizer in TorchScript. 
2022-05-18T03:55:14.5268642Z On WorkerInfo(id=0, name=worker0):
2022-05-18T03:55:14.5269203Z ValueError('Error creating optimizer.')
2022-05-18T03:55:14.5269544Z Traceback (most recent call last):
2022-05-18T03:55:14.5270205Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function
2022-05-18T03:55:14.5270782Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-05-18T03:55:14.5271575Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 90, in _new_local_optimizer
2022-05-18T03:55:14.5272168Z     _LocalOptimizer(optim_cls, local_params_rref, *args, **kwargs))
2022-05-18T03:55:14.5272891Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 77, in __init__
2022-05-18T03:55:14.5273351Z     **kwargs)
2022-05-18T03:55:14.5274054Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/dist_optimizer_test.py", line 45, in __init__
2022-05-18T03:55:14.5274613Z     raise ValueError("Error creating optimizer.")
2022-05-18T03:55:14.5275017Z ValueError: Error creating optimizer.
2022-05-18T03:55:14.5275244Z 
2022-05-18T03:55:14.5303734Z On WorkerInfo(id=0, name=worker0):
2022-05-18T03:55:14.5304329Z ValueError('Error creating optimizer.')
2022-05-18T03:55:14.5304734Z Traceback (most recent call last):
2022-05-18T03:55:14.5305425Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function
2022-05-18T03:55:14.5305994Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-05-18T03:55:14.5306753Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 90, in _new_local_optimizer
2022-05-18T03:55:14.5307345Z     _LocalOptimizer(optim_cls, local_params_rref, *args, **kwargs))
2022-05-18T03:55:14.5308071Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 77, in __init__
2022-05-18T03:55:14.5308500Z     **kwargs)
2022-05-18T03:55:14.5309189Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/dist_optimizer_test.py", line 45, in __init__
2022-05-18T03:55:14.5309758Z     raise ValueError("Error creating optimizer.")
2022-05-18T03:55:14.5310154Z ValueError: Error creating optimizer.
2022-05-18T03:55:14.5310388Z 
2022-05-18T03:55:14.5404467Z On WorkerInfo(id=2, name=worker2):
2022-05-18T03:55:14.5405046Z ValueError('Error creating optimizer.')
2022-05-18T03:55:14.5405443Z Traceback (most recent call last):
2022-05-18T03:55:14.5406079Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function
2022-05-18T03:55:14.5406539Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-05-18T03:55:14.5407517Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 90, in _new_local_optimizer
2022-05-18T03:55:14.5408275Z     _LocalOptimizer(optim_cls, local_params_rref, *args, **kwargs))
2022-05-18T03:55:14.5409193Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 77, in __init__
2022-05-18T03:55:14.5409943Z     **kwargs)
2022-05-18T03:55:14.5410732Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/dist_optimizer_test.py", line 45, in __init__
2022-05-18T03:55:14.5411342Z     raise ValueError("Error creating optimizer.")
2022-05-18T03:55:14.5411752Z ValueError: Error creating optimizer.
2022-05-18T03:55:14.5412133Z 
2022-05-18T03:55:14.5629570Z On WorkerInfo(id=3, name=worker3):
2022-05-18T03:55:14.5630194Z ValueError('Error creating optimizer.')
2022-05-18T03:55:14.5630644Z Traceback (most recent call last):
2022-05-18T03:55:14.5632583Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function
2022-05-18T03:55:14.5633199Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-05-18T03:55:14.5634052Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 90, in _new_local_optimizer
2022-05-18T03:55:14.5634698Z     _LocalOptimizer(optim_cls, local_params_rref, *args, **kwargs))
2022-05-18T03:55:14.5635138Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 77, in __init__
2022-05-18T03:55:14.5635404Z     **kwargs)
2022-05-18T03:55:14.5635794Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/dist_optimizer_test.py", line 45, in __init__
2022-05-18T03:55:14.5636139Z     raise ValueError("Error creating optimizer.")
2022-05-18T03:55:14.5636456Z ValueError: Error creating optimizer.
2022-05-18T03:55:14.5636592Z 
2022-05-18T03:55:14.5637213Z On WorkerInfo(id=3, name=worker3):
2022-05-18T03:55:14.5637579Z ValueError('Error creating optimizer.')
2022-05-18T03:55:14.5637942Z Traceback (most recent call last):
2022-05-18T03:55:14.5638656Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function
2022-05-18T03:55:14.5639268Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-05-18T03:55:14.5640061Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 90, in _new_local_optimizer
2022-05-18T03:55:14.5640412Z     _LocalOptimizer(optim_cls, local_params_rref, *args, **kwargs))
2022-05-18T03:55:14.5640824Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 77, in __init__
2022-05-18T03:55:14.5641084Z     **kwargs)
2022-05-18T03:55:14.5641463Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/dist_optimizer_test.py", line 45, in __init__
2022-05-18T03:55:14.5641793Z     raise ValueError("Error creating optimizer.")
2022-05-18T03:55:14.5642028Z ValueError: Error creating optimizer.
2022-05-18T03:55:14.5642161Z 
2022-05-18T03:55:14.5734242Z On WorkerInfo(id=1, name=worker1):
2022-05-18T03:55:14.5734759Z ValueError('Error creating optimizer.')
2022-05-18T03:55:14.5735197Z Traceback (most recent call last):
2022-05-18T03:55:14.5735953Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function
2022-05-18T03:55:14.5736651Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-05-18T03:55:14.5737254Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 90, in _new_local_optimizer
2022-05-18T03:55:14.5737599Z     _LocalOptimizer(optim_cls, local_params_rref, *args, **kwargs))
2022-05-18T03:55:14.5738012Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/optim/optimizer.py", line 77, in __init__
2022-05-18T03:55:14.5738267Z     **kwargs)
2022-05-18T03:55:14.5738659Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/dist_optimizer_test.py", line 45, in __init__
2022-05-18T03:55:14.5738989Z     raise ValueError("Error creating optimizer.")
2022-05-18T03:55:14.5739211Z ValueError: Error creating optimizer.
2022-05-18T03:55:14.5739342Z 
2022-05-18T03:55:14.8050710Z ok (1.732s)
2022-05-18T03:55:14.8051114Z 
2022-05-18T03:55:14.8051429Z ----------------------------------------------------------------------
2022-05-18T03:55:14.8051759Z Ran 1 test in 1.733s
2022-05-18T03:55:14.8051876Z 
2022-05-18T03:55:14.8051992Z OK
2022-05-18T03:55:14.8052085Z 
2022-05-18T03:55:14.8052179Z Generating XML reports...
2022-05-18T03:55:14.8084901Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistOptimizerTest-20220518035513.xml
2022-05-18T03:55:15.5842223Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf8x_piqy
2022-05-18T03:55:15.5842911Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf8x_piqy/_remote_module_non_scriptable.py
2022-05-18T03:55:15.8378448Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T03:55:15.8388069Z 
2022-05-18T03:55:15.8388177Z Running tests...
2022-05-18T03:55:15.8388687Z ----------------------------------------------------------------------
2022-05-18T03:55:16.1461240Z   test_dist_optim_none_grads (__main__.TensorPipeDistOptimizerTest) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77075 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure IN_CI is not set and you are not using the flag --import-disabled-tests. (0.307s)
2022-05-18T03:55:16.1461752Z 
2022-05-18T03:55:16.1461962Z ----------------------------------------------------------------------
2022-05-18T03:55:16.1462212Z Ran 1 test in 0.307s
2022-05-18T03:55:16.1462326Z 
2022-05-18T03:55:16.1462398Z OK (skipped=1)
2022-05-18T03:55:16.1462505Z 
2022-05-18T03:55:16.1462590Z Generating XML reports...
2022-05-18T03:55:16.1484752Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistOptimizerTest-20220518035515.xml
2022-05-18T03:55:16.8611108Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2k4g6zln
2022-05-18T03:55:16.8611802Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2k4g6zln/_remote_module_non_scriptable.py
2022-05-18T03:55:17.1131350Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T03:55:17.1141026Z 
2022-05-18T03:55:17.1141121Z Running tests...
2022-05-18T03:55:17.1141574Z ----------------------------------------------------------------------
2022-05-18T03:55:17.4295670Z   test_dist_backward (__main__.TensorPipeJitDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23689
2022-05-18T03:55:17.4318023Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23690
2022-05-18T03:55:17.4340488Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23691
2022-05-18T03:55:17.4365558Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 23692
2022-05-18T03:55:18.0614520Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcn_btozl
2022-05-18T03:55:18.0615367Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcn_btozl/_remote_module_non_scriptable.py
2022-05-18T03:55:18.0740235Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyw7vqjje
2022-05-18T03:55:18.0741072Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyw7vqjje/_remote_module_non_scriptable.py
2022-05-18T03:55:18.0800621Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpiqswfsj3
2022-05-18T03:55:18.0801988Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpiqswfsj3/_remote_module_non_scriptable.py
2022-05-18T03:55:18.0923398Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq2w_gptu
2022-05-18T03:55:18.0924653Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq2w_gptu/_remote_module_non_scriptable.py
2022-05-18T03:55:18.3116530Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:55:18.3233906Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:55:18.3289374Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:55:18.3431993Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:55:18.8404712Z ok (1.726s)
2022-05-18T03:55:18.8405000Z 
2022-05-18T03:55:18.8405447Z ----------------------------------------------------------------------
2022-05-18T03:55:18.8405683Z Ran 1 test in 1.726s
2022-05-18T03:55:18.8405798Z 
2022-05-18T03:55:18.8405861Z OK
2022-05-18T03:55:18.8405954Z 
2022-05-18T03:55:18.8406048Z Generating XML reports...
2022-05-18T03:55:18.8440068Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitDistAutogradTest-20220518035517.xml
2022-05-18T03:55:19.6479517Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf7v57jd_
2022-05-18T03:55:19.6480544Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf7v57jd_/_remote_module_non_scriptable.py
2022-05-18T03:55:19.9055394Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T03:55:19.9065006Z 
2022-05-18T03:55:19.9065121Z Running tests...
2022-05-18T03:55:19.9065533Z ----------------------------------------------------------------------
2022-05-18T03:55:20.2319710Z   test_get_gradients (__main__.TensorPipeJitDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23918
2022-05-18T03:55:20.2341456Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23919
2022-05-18T03:55:20.2365349Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23920
2022-05-18T03:55:20.2390145Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 23921
2022-05-18T03:55:20.8148534Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplo25j7u5
2022-05-18T03:55:20.8149812Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplo25j7u5/_remote_module_non_scriptable.py
2022-05-18T03:55:20.8160448Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbfnztxlj
2022-05-18T03:55:20.8162327Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbfnztxlj/_remote_module_non_scriptable.py
2022-05-18T03:55:20.8704434Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6irwkna5
2022-05-18T03:55:20.8705230Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6irwkna5/_remote_module_non_scriptable.py
2022-05-18T03:55:20.8750314Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp207t_8tq
2022-05-18T03:55:20.8752351Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp207t_8tq/_remote_module_non_scriptable.py
2022-05-18T03:55:21.0628653Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:55:21.0635648Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:55:21.1201048Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:55:21.1238620Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:55:21.6431945Z ok (1.736s)
2022-05-18T03:55:21.6432101Z 
2022-05-18T03:55:21.6432413Z ----------------------------------------------------------------------
2022-05-18T03:55:21.6432668Z Ran 1 test in 1.737s
2022-05-18T03:55:21.6432786Z 
2022-05-18T03:55:21.6432846Z OK
2022-05-18T03:55:21.6432937Z 
2022-05-18T03:55:21.6433033Z Generating XML reports...
2022-05-18T03:55:21.6468509Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitDistAutogradTest-20220518035519.xml
2022-05-18T03:55:22.4425729Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnfx2p8r2
2022-05-18T03:55:22.4427020Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnfx2p8r2/_remote_module_non_scriptable.py
2022-05-18T03:55:22.6953885Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T03:55:22.6963933Z 
2022-05-18T03:55:22.6964321Z Running tests...
2022-05-18T03:55:23.0086817Z ----------------------------------------------------------------------
2022-05-18T03:55:23.0087564Z   test_jit_fork_within_context (__main__.TensorPipeJitDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24157
2022-05-18T03:55:23.0109379Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24158
2022-05-18T03:55:23.0132636Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24159
2022-05-18T03:55:23.0155987Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 24160
2022-05-18T03:55:23.6340484Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp09v25zxh
2022-05-18T03:55:23.6341340Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp09v25zxh/_remote_module_non_scriptable.py
2022-05-18T03:55:23.6415774Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg3q5zyo3
2022-05-18T03:55:23.6417118Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg3q5zyo3/_remote_module_non_scriptable.py
2022-05-18T03:55:23.6421223Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp__91hro9
2022-05-18T03:55:23.6424005Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp__91hro9/_remote_module_non_scriptable.py
2022-05-18T03:55:23.6443872Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps9tgb21m
2022-05-18T03:55:23.6445882Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps9tgb21m/_remote_module_non_scriptable.py
2022-05-18T03:55:23.8825573Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:55:23.8868282Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:55:23.8891157Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:55:23.8920377Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:55:24.5198810Z ok (1.823s)
2022-05-18T03:55:24.5199196Z 
2022-05-18T03:55:24.5199545Z ----------------------------------------------------------------------
2022-05-18T03:55:24.5199835Z Ran 1 test in 1.823s
2022-05-18T03:55:24.5200000Z 
2022-05-18T03:55:24.5200068Z OK
2022-05-18T03:55:24.5200160Z 
2022-05-18T03:55:24.5200257Z Generating XML reports...
2022-05-18T03:55:24.5235288Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitDistAutogradTest-20220518035522.xml
2022-05-18T03:55:25.2997700Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpniy153k_
2022-05-18T03:55:25.2998238Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpniy153k_/_remote_module_non_scriptable.py
2022-05-18T03:55:25.5534157Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T03:55:25.5544149Z 
2022-05-18T03:55:25.5544291Z Running tests...
2022-05-18T03:55:25.5544745Z ----------------------------------------------------------------------
2022-05-18T03:55:25.8654590Z   test_restore_context_after_swtich_to_jit_thread (__main__.TensorPipeJitDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24396
2022-05-18T03:55:25.8678112Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24397
2022-05-18T03:55:25.8701399Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24398
2022-05-18T03:55:25.8725160Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 24399
2022-05-18T03:55:26.5466973Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvqhlut1g
2022-05-18T03:55:26.5468362Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvqhlut1g/_remote_module_non_scriptable.py
2022-05-18T03:55:26.5863956Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb1o_yynm
2022-05-18T03:55:26.5864733Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb1o_yynm/_remote_module_non_scriptable.py
2022-05-18T03:55:26.6135795Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvqfp7mwr
2022-05-18T03:55:26.6137152Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvqfp7mwr/_remote_module_non_scriptable.py
2022-05-18T03:55:26.6148102Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjab4_7xo
2022-05-18T03:55:26.6150567Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjab4_7xo/_remote_module_non_scriptable.py
2022-05-18T03:55:26.7947177Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:55:26.8328944Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:55:26.8588502Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:55:26.8634552Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:55:27.3767531Z ok (1.822s)
2022-05-18T03:55:27.3768005Z 
2022-05-18T03:55:27.3768598Z ----------------------------------------------------------------------
2022-05-18T03:55:27.3769038Z Ran 1 test in 1.822s
2022-05-18T03:55:27.3769240Z 
2022-05-18T03:55:27.3769331Z OK
2022-05-18T03:55:27.3769470Z 
2022-05-18T03:55:27.3769606Z Generating XML reports...
2022-05-18T03:55:27.3803799Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitDistAutogradTest-20220518035525.xml
2022-05-18T03:55:28.1521283Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmy9trhfu
2022-05-18T03:55:28.1522338Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmy9trhfu/_remote_module_non_scriptable.py
2022-05-18T03:55:28.4050061Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T03:55:28.4059947Z 
2022-05-18T03:55:28.4060355Z Running tests...
2022-05-18T03:55:28.4061011Z ----------------------------------------------------------------------
2022-05-18T03:55:28.7227606Z   test_add_done_callback (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24625
2022-05-18T03:55:28.7249908Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24626
2022-05-18T03:55:28.7273055Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24627
2022-05-18T03:55:28.7296933Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 24628
2022-05-18T03:55:29.3719079Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgm6538ri
2022-05-18T03:55:29.3720257Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgm6538ri/_remote_module_non_scriptable.py
2022-05-18T03:55:29.3750624Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplnim7_x6
2022-05-18T03:55:29.3752921Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplnim7_x6/_remote_module_non_scriptable.py
2022-05-18T03:55:29.3869329Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy184edzm
2022-05-18T03:55:29.3871470Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy184edzm/_remote_module_non_scriptable.py
2022-05-18T03:55:29.3977336Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeppirb2y
2022-05-18T03:55:29.3978424Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeppirb2y/_remote_module_non_scriptable.py
2022-05-18T03:55:29.6205094Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:55:29.6207369Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:55:29.6366873Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:55:29.6466524Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:55:30.2339723Z ok (1.828s)
2022-05-18T03:55:30.2339935Z 
2022-05-18T03:55:30.2340316Z ----------------------------------------------------------------------
2022-05-18T03:55:30.2340565Z Ran 1 test in 1.828s
2022-05-18T03:55:30.2340668Z 
2022-05-18T03:55:30.2340729Z OK
2022-05-18T03:55:30.2340821Z 
2022-05-18T03:55:30.2340967Z Generating XML reports...
2022-05-18T03:55:30.2374588Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220518035528.xml
2022-05-18T03:55:31.0053385Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9tfmurcj
2022-05-18T03:55:31.0054213Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9tfmurcj/_remote_module_non_scriptable.py
2022-05-18T03:55:31.2570997Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T03:55:31.2579533Z 
2022-05-18T03:55:31.2579671Z Running tests...
2022-05-18T03:55:31.2580653Z ----------------------------------------------------------------------
2022-05-18T03:55:31.5703577Z   test_all_kwargs_are_populated_by_defaults (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24860
2022-05-18T03:55:31.5727634Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24861
2022-05-18T03:55:31.5750881Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24862
2022-05-18T03:55:31.5774376Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 24863
2022-05-18T03:55:32.2700413Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdu217kdu
2022-05-18T03:55:32.2701556Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdu217kdu/_remote_module_non_scriptable.py
2022-05-18T03:55:32.2780651Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjaudou41
2022-05-18T03:55:32.2781683Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjaudou41/_remote_module_non_scriptable.py
2022-05-18T03:55:32.2978045Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp80g5v2k8
2022-05-18T03:55:32.2978909Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp80g5v2k8/_remote_module_non_scriptable.py
2022-05-18T03:55:32.3482322Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgudi5wiw
2022-05-18T03:55:32.3483423Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgudi5wiw/_remote_module_non_scriptable.py
2022-05-18T03:55:32.5202728Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:55:32.5289003Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:55:32.5459138Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:55:32.5962527Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:55:33.1818433Z ok (1.924s)
2022-05-18T03:55:33.1818706Z 
2022-05-18T03:55:33.1819061Z ----------------------------------------------------------------------
2022-05-18T03:55:33.1819313Z Ran 1 test in 1.924s
2022-05-18T03:55:33.1819429Z 
2022-05-18T03:55:33.1819525Z OK
2022-05-18T03:55:33.1819636Z 
2022-05-18T03:55:33.1819734Z Generating XML reports...
2022-05-18T03:55:33.1853576Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220518035531.xml
2022-05-18T03:55:33.9542177Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw9neqd4u
2022-05-18T03:55:33.9543066Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw9neqd4u/_remote_module_non_scriptable.py
2022-05-18T03:55:34.2061561Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T03:55:34.2072337Z 
2022-05-18T03:55:34.2072422Z Running tests...
2022-05-18T03:55:34.2073349Z ----------------------------------------------------------------------
2022-05-18T03:55:34.5235800Z   test_args_and_kwargs_contain_different_types (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25083
2022-05-18T03:55:34.5257717Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25084
2022-05-18T03:55:34.5281331Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25085
2022-05-18T03:55:34.5305684Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 25086
2022-05-18T03:55:35.1127565Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe30yip99
2022-05-18T03:55:35.1128351Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe30yip99/_remote_module_non_scriptable.py
2022-05-18T03:55:35.1132134Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdtaub58n
2022-05-18T03:55:35.1134161Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdtaub58n/_remote_module_non_scriptable.py
2022-05-18T03:55:35.1474103Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqx0amx7z
2022-05-18T03:55:35.1474876Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqx0amx7z/_remote_module_non_scriptable.py
2022-05-18T03:55:35.1817335Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxo_xk0xd
2022-05-18T03:55:35.1818120Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxo_xk0xd/_remote_module_non_scriptable.py
2022-05-18T03:55:35.3597182Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:55:35.3606430Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:55:35.3941850Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:55:35.4294433Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:55:36.0348623Z ok (1.827s)
2022-05-18T03:55:36.0348856Z 
2022-05-18T03:55:36.0349371Z ----------------------------------------------------------------------
2022-05-18T03:55:36.0349830Z Ran 1 test in 1.828s
2022-05-18T03:55:36.0349950Z 
2022-05-18T03:55:36.0350011Z OK
2022-05-18T03:55:36.0350104Z 
2022-05-18T03:55:36.0350184Z Generating XML reports...
2022-05-18T03:55:36.0383980Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220518035534.xml
2022-05-18T03:55:36.8060289Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph1wlj018
2022-05-18T03:55:36.8061443Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph1wlj018/_remote_module_non_scriptable.py
2022-05-18T03:55:37.0589607Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T03:55:37.0600094Z 
2022-05-18T03:55:37.0600440Z Running tests...
2022-05-18T03:55:37.0600864Z ----------------------------------------------------------------------
2022-05-18T03:55:37.3775093Z   test_args_kwargs_are_neither_passed (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25306
2022-05-18T03:55:37.3797828Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25307
2022-05-18T03:55:37.3821281Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25308
2022-05-18T03:55:37.3845543Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 25309
2022-05-18T03:55:38.0526534Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgtxnh62x
2022-05-18T03:55:38.0527358Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgtxnh62x/_remote_module_non_scriptable.py
2022-05-18T03:55:38.0887216Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8_9xjdzv
2022-05-18T03:55:38.0888083Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8_9xjdzv/_remote_module_non_scriptable.py
2022-05-18T03:55:38.1067406Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpoafuqlku
2022-05-18T03:55:38.1068444Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpoafuqlku/_remote_module_non_scriptable.py
2022-05-18T03:55:38.1160746Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp175fnmud
2022-05-18T03:55:38.1162195Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp175fnmud/_remote_module_non_scriptable.py
2022-05-18T03:55:38.3020981Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:55:38.3355528Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:55:38.3539392Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:55:38.3640043Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:55:38.8887861Z ok (1.828s)
2022-05-18T03:55:38.8888127Z 
2022-05-18T03:55:38.8888678Z ----------------------------------------------------------------------
2022-05-18T03:55:38.8889029Z Ran 1 test in 1.829s
2022-05-18T03:55:38.8889145Z 
2022-05-18T03:55:38.8889206Z OK
2022-05-18T03:55:38.8889285Z 
2022-05-18T03:55:38.8889383Z Generating XML reports...
2022-05-18T03:55:38.8923417Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220518035537.xml
2022-05-18T03:55:39.6606274Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3e8w0cz1
2022-05-18T03:55:39.6607147Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3e8w0cz1/_remote_module_non_scriptable.py
2022-05-18T03:55:39.9177643Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T03:55:39.9187007Z 
2022-05-18T03:55:39.9187145Z Running tests...
2022-05-18T03:55:39.9188066Z ----------------------------------------------------------------------
2022-05-18T03:55:40.2361864Z   test_async_function_remote (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25529
2022-05-18T03:55:40.2385424Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25530
2022-05-18T03:55:40.2408468Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25531
2022-05-18T03:55:40.2433544Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 25532
2022-05-18T03:55:40.8759331Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqna72zhf
2022-05-18T03:55:40.8760082Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqna72zhf/_remote_module_non_scriptable.py
2022-05-18T03:55:40.8775969Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp88nn9ytb
2022-05-18T03:55:40.8777758Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp88nn9ytb/_remote_module_non_scriptable.py
2022-05-18T03:55:40.8898475Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpow504gkt
2022-05-18T03:55:40.8899580Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpow504gkt/_remote_module_non_scriptable.py
2022-05-18T03:55:40.8953622Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwmjjyaxn
2022-05-18T03:55:40.8954553Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwmjjyaxn/_remote_module_non_scriptable.py
2022-05-18T03:55:41.1229546Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:55:41.1235319Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:55:41.1384426Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:55:41.1419800Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:55:41.7476327Z ok (1.829s)
2022-05-18T03:55:41.7476539Z 
2022-05-18T03:55:41.7478067Z ----------------------------------------------------------------------
2022-05-18T03:55:41.7478588Z Ran 1 test in 1.829s
2022-05-18T03:55:41.7478803Z 
2022-05-18T03:55:41.7478895Z OK
2022-05-18T03:55:41.7478977Z 
2022-05-18T03:55:41.7479071Z Generating XML reports...
2022-05-18T03:55:41.7511644Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220518035539.xml
2022-05-18T03:55:42.5204370Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy4k9_gkd
2022-05-18T03:55:42.5204975Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy4k9_gkd/_remote_module_non_scriptable.py
2022-05-18T03:55:42.7750294Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T03:55:42.7759912Z 
2022-05-18T03:55:42.7760006Z Running tests...
2022-05-18T03:55:42.7760744Z ----------------------------------------------------------------------
2022-05-18T03:55:43.0908792Z   test_async_function_remote_multi (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25748
2022-05-18T03:55:43.0930085Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25749
2022-05-18T03:55:43.0953304Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25750
2022-05-18T03:55:43.0976921Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 25751
2022-05-18T03:55:43.7419747Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz91_wxn2
2022-05-18T03:55:43.7420539Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz91_wxn2/_remote_module_non_scriptable.py
2022-05-18T03:55:43.7449348Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp781llql_
2022-05-18T03:55:43.7450860Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp781llql_/_remote_module_non_scriptable.py
2022-05-18T03:55:43.7564238Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv331tsbq
2022-05-18T03:55:43.7565434Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv331tsbq/_remote_module_non_scriptable.py
2022-05-18T03:55:43.7567318Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6jqtqx4h
2022-05-18T03:55:43.7570592Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6jqtqx4h/_remote_module_non_scriptable.py
2022-05-18T03:55:43.9883944Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:55:43.9921268Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:55:44.0016520Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:55:44.0025426Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:55:44.6048666Z ok (1.829s)
2022-05-18T03:55:44.6048865Z 
2022-05-18T03:55:44.6049439Z ----------------------------------------------------------------------
2022-05-18T03:55:44.6049771Z Ran 1 test in 1.829s
2022-05-18T03:55:44.6049885Z 
2022-05-18T03:55:44.6049947Z OK
2022-05-18T03:55:44.6050037Z 
2022-05-18T03:55:44.6050118Z Generating XML reports...
2022-05-18T03:55:44.6083899Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220518035542.xml
2022-05-18T03:55:45.3726355Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfzwipcq6
2022-05-18T03:55:45.3727359Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfzwipcq6/_remote_module_non_scriptable.py
2022-05-18T03:55:45.6253046Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T03:55:45.6262568Z 
2022-05-18T03:55:45.6262675Z Running tests...
2022-05-18T03:55:45.6263458Z ----------------------------------------------------------------------
2022-05-18T03:55:45.9391384Z   test_async_function_simple (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25967
2022-05-18T03:55:45.9414312Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25968
2022-05-18T03:55:45.9437579Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25969
2022-05-18T03:55:45.9461830Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 25970
2022-05-18T03:55:46.6125987Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnq00qbc3
2022-05-18T03:55:46.6127122Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnq00qbc3/_remote_module_non_scriptable.py
2022-05-18T03:55:46.6458499Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptr2g72j4
2022-05-18T03:55:46.6459636Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptr2g72j4/_remote_module_non_scriptable.py
2022-05-18T03:55:46.6493954Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkkpt9w4j
2022-05-18T03:55:46.6494818Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkkpt9w4j/_remote_module_non_scriptable.py
2022-05-18T03:55:46.6692945Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpot0cb9qd
2022-05-18T03:55:46.6694560Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpot0cb9qd/_remote_module_non_scriptable.py
2022-05-18T03:55:46.8614153Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:55:46.8913498Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:55:46.8962901Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:55:46.9167383Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:55:47.4503782Z ok (1.824s)
2022-05-18T03:55:47.4504159Z 
2022-05-18T03:55:47.4504840Z ----------------------------------------------------------------------
2022-05-18T03:55:47.4505275Z Ran 1 test in 1.824s
2022-05-18T03:55:47.4505404Z 
2022-05-18T03:55:47.4505466Z OK
2022-05-18T03:55:47.4505545Z 
2022-05-18T03:55:47.4505638Z Generating XML reports...
2022-05-18T03:55:47.4539749Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220518035545.xml
2022-05-18T03:55:48.2242066Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkhn5uwiu
2022-05-18T03:55:48.2242553Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkhn5uwiu/_remote_module_non_scriptable.py
2022-05-18T03:55:48.4772568Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T03:55:48.4782563Z 
2022-05-18T03:55:48.4782668Z Running tests...
2022-05-18T03:55:48.4783484Z ----------------------------------------------------------------------
2022-05-18T03:55:48.7956650Z   test_async_function_wrong_decorator_order (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26186
2022-05-18T03:55:48.7979102Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26187
2022-05-18T03:55:48.8002009Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26188
2022-05-18T03:55:48.8026652Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 26189
2022-05-18T03:55:49.4588378Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdtq4p_ts
2022-05-18T03:55:49.4589101Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdtq4p_ts/_remote_module_non_scriptable.py
2022-05-18T03:55:49.5163067Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc6v4tmrc
2022-05-18T03:55:49.5163887Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc6v4tmrc/_remote_module_non_scriptable.py
2022-05-18T03:55:49.5234124Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpby_jvur_
2022-05-18T03:55:49.5236572Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpby_jvur_/_remote_module_non_scriptable.py
2022-05-18T03:55:49.5365101Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2s1qeo1k
2022-05-18T03:55:49.5366133Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2s1qeo1k/_remote_module_non_scriptable.py
2022-05-18T03:55:49.7097662Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:55:49.7619068Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:55:49.7714407Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:55:49.7825807Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:55:50.2067363Z ok (1.728s)
2022-05-18T03:55:50.2067612Z 
2022-05-18T03:55:50.2068155Z ----------------------------------------------------------------------
2022-05-18T03:55:50.2068511Z Ran 1 test in 1.728s
2022-05-18T03:55:50.2068613Z 
2022-05-18T03:55:50.2068673Z OK
2022-05-18T03:55:50.2068764Z 
2022-05-18T03:55:50.2068860Z Generating XML reports...
2022-05-18T03:55:50.2101789Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220518035548.xml
2022-05-18T03:55:50.9742743Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqivjvf4k
2022-05-18T03:55:50.9743826Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqivjvf4k/_remote_module_non_scriptable.py
2022-05-18T03:55:51.2276573Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T03:55:51.2286578Z 
2022-05-18T03:55:51.2286918Z Running tests...
2022-05-18T03:55:51.2287283Z ----------------------------------------------------------------------
2022-05-18T03:55:51.5428304Z   test_async_function_wrong_return_type (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26405
2022-05-18T03:55:51.5450696Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26406
2022-05-18T03:55:51.5473892Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26407
2022-05-18T03:55:51.5497642Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 26408
2022-05-18T03:55:52.1368312Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpugjvb8tm
2022-05-18T03:55:52.1369096Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpugjvb8tm/_remote_module_non_scriptable.py
2022-05-18T03:55:52.1728939Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc1epeibo
2022-05-18T03:55:52.1730037Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc1epeibo/_remote_module_non_scriptable.py
2022-05-18T03:55:52.1777563Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj7qlqgj3
2022-05-18T03:55:52.1778782Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj7qlqgj3/_remote_module_non_scriptable.py
2022-05-18T03:55:52.1842106Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppj__tvar
2022-05-18T03:55:52.1843618Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppj__tvar/_remote_module_non_scriptable.py
2022-05-18T03:55:52.3860606Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:55:52.4220714Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:55:52.4286711Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:55:52.4342331Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:55:52.9537885Z ok (1.725s)
2022-05-18T03:55:52.9538138Z 
2022-05-18T03:55:52.9538651Z ----------------------------------------------------------------------
2022-05-18T03:55:52.9539028Z Ran 1 test in 1.725s
2022-05-18T03:55:52.9539130Z 
2022-05-18T03:55:52.9539190Z OK
2022-05-18T03:55:52.9539280Z 
2022-05-18T03:55:52.9539373Z Generating XML reports...
2022-05-18T03:55:52.9574963Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220518035551.xml
2022-05-18T03:55:53.7220729Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9zpnt_f6
2022-05-18T03:55:53.7221289Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9zpnt_f6/_remote_module_non_scriptable.py
2022-05-18T03:55:53.9734974Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T03:55:53.9745105Z 
2022-05-18T03:55:53.9745535Z Running tests...
2022-05-18T03:55:53.9745933Z ----------------------------------------------------------------------
2022-05-18T03:55:54.2875335Z   test_async_function_wrong_return_type_remote (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26624
2022-05-18T03:55:54.2896963Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26625
2022-05-18T03:55:54.2920427Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26626
2022-05-18T03:55:54.2944322Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 26627
2022-05-18T03:55:55.0084094Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphecjsio2
2022-05-18T03:55:55.0084803Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfk4pezrw
2022-05-18T03:55:55.0085498Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphecjsio2/_remote_module_non_scriptable.py
2022-05-18T03:55:55.0086062Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfk4pezrw/_remote_module_non_scriptable.py
2022-05-18T03:55:55.0160747Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4_f2r63f
2022-05-18T03:55:55.0162288Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4_f2r63f/_remote_module_non_scriptable.py
2022-05-18T03:55:55.0471794Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppwpn6ej7
2022-05-18T03:55:55.0472779Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppwpn6ej7/_remote_module_non_scriptable.py
2022-05-18T03:55:55.2565151Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:55:55.2565600Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:55:55.2663524Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:55:55.2935490Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:55:55.7057219Z ok (1.731s)
2022-05-18T03:55:55.7057391Z 
2022-05-18T03:55:55.7057693Z ----------------------------------------------------------------------
2022-05-18T03:55:55.7057980Z Ran 1 test in 1.731s
2022-05-18T03:55:55.7058095Z 
2022-05-18T03:55:55.7058155Z OK
2022-05-18T03:55:55.7058244Z 
2022-05-18T03:55:55.7058339Z Generating XML reports...
2022-05-18T03:55:55.7092220Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220518035553.xml
2022-05-18T03:55:56.4695809Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8wlrhufy
2022-05-18T03:55:56.4696648Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8wlrhufy/_remote_module_non_scriptable.py
2022-05-18T03:55:56.7228308Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T03:55:56.7237926Z 
2022-05-18T03:55:56.7238056Z Running tests...
2022-05-18T03:55:56.7238498Z ----------------------------------------------------------------------
2022-05-18T03:55:57.0359581Z   test_async_script_throw (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26843
2022-05-18T03:55:57.0383789Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26844
2022-05-18T03:55:57.0407066Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26845
2022-05-18T03:55:57.0432614Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 26846
2022-05-18T03:55:57.7295251Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7u3mr59h
2022-05-18T03:55:57.7296085Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7u3mr59h/_remote_module_non_scriptable.py
2022-05-18T03:55:57.7437854Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp19q8d059
2022-05-18T03:55:57.7440851Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp19q8d059/_remote_module_non_scriptable.py
2022-05-18T03:55:57.7717432Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuu1g_3p6
2022-05-18T03:55:57.7718749Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuu1g_3p6/_remote_module_non_scriptable.py
2022-05-18T03:55:57.7729735Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0hh94jyv
2022-05-18T03:55:57.7742696Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0hh94jyv/_remote_module_non_scriptable.py
2022-05-18T03:55:57.9772110Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:55:57.9914893Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:55:58.0186972Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:55:58.0210035Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:55:58.5475329Z ok (1.823s)
2022-05-18T03:55:58.5475541Z 
2022-05-18T03:55:58.5475917Z ----------------------------------------------------------------------
2022-05-18T03:55:58.5476389Z Ran 1 test in 1.824s
2022-05-18T03:55:58.5476599Z 
2022-05-18T03:55:58.5476696Z OK
2022-05-18T03:55:58.5476777Z 
2022-05-18T03:55:58.5476873Z Generating XML reports...
2022-05-18T03:55:58.5510889Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220518035556.xml
2022-05-18T03:55:59.3187490Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpld549mkp
2022-05-18T03:55:59.3188262Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpld549mkp/_remote_module_non_scriptable.py
2022-05-18T03:55:59.5715629Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T03:55:59.5725416Z 
2022-05-18T03:55:59.5725796Z Running tests...
2022-05-18T03:55:59.5726244Z ----------------------------------------------------------------------
2022-05-18T03:55:59.8864532Z   test_async_script_udf (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27078
2022-05-18T03:55:59.8887592Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27079
2022-05-18T03:55:59.8911139Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27080
2022-05-18T03:55:59.8935047Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 27081
2022-05-18T03:56:00.4916842Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5u2yeici
2022-05-18T03:56:00.4917651Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5u2yeici/_remote_module_non_scriptable.py
2022-05-18T03:56:00.5029390Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpchyt0kjp
2022-05-18T03:56:00.5030359Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpchyt0kjp/_remote_module_non_scriptable.py
2022-05-18T03:56:00.5308162Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpepxzw00u
2022-05-18T03:56:00.5309318Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpepxzw00u/_remote_module_non_scriptable.py
2022-05-18T03:56:00.5412531Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdjnu_w3x
2022-05-18T03:56:00.5413342Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdjnu_w3x/_remote_module_non_scriptable.py
2022-05-18T03:56:00.7409773Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:56:00.7514636Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:56:00.7808688Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:56:00.7906535Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:56:01.2974307Z ok (1.725s)
2022-05-18T03:56:01.2974581Z 
2022-05-18T03:56:01.2975067Z ----------------------------------------------------------------------
2022-05-18T03:56:01.2975522Z Ran 1 test in 1.725s
2022-05-18T03:56:01.2975697Z 
2022-05-18T03:56:01.2975765Z OK
2022-05-18T03:56:01.2975902Z 
2022-05-18T03:56:01.2975999Z Generating XML reports...
2022-05-18T03:56:01.3008922Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220518035559.xml
2022-05-18T03:56:02.0793761Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpto0aubbh
2022-05-18T03:56:02.0794515Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpto0aubbh/_remote_module_non_scriptable.py
2022-05-18T03:56:02.3345434Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T03:56:02.3354871Z 
2022-05-18T03:56:02.3354976Z Running tests...
2022-05-18T03:56:02.3355869Z ----------------------------------------------------------------------
2022-05-18T03:56:02.6522545Z   test_call_fork_in_jit_with_profiling (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27313
2022-05-18T03:56:02.6546270Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27314
2022-05-18T03:56:02.6569016Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27315
2022-05-18T03:56:02.6593235Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 27316
2022-05-18T03:56:03.2791404Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpckvycx9g
2022-05-18T03:56:03.2792277Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpckvycx9g/_remote_module_non_scriptable.py
2022-05-18T03:56:03.2848073Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv02c50w_
2022-05-18T03:56:03.2849586Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv02c50w_/_remote_module_non_scriptable.py
2022-05-18T03:56:03.3133043Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0l6proq1
2022-05-18T03:56:03.3134436Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0l6proq1/_remote_module_non_scriptable.py
2022-05-18T03:56:03.3229749Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp4q8evek
2022-05-18T03:56:03.3231244Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp4q8evek/_remote_module_non_scriptable.py
2022-05-18T03:56:03.5273651Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:56:03.5321727Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:56:03.5631216Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:56:03.5699591Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:56:03.7628606Z ok (1.427s)
2022-05-18T03:56:03.7628866Z 
2022-05-18T03:56:03.7629316Z ----------------------------------------------------------------------
2022-05-18T03:56:03.7629566Z Ran 1 test in 1.427s
2022-05-18T03:56:03.7629669Z 
2022-05-18T03:56:03.7629730Z OK
2022-05-18T03:56:03.7629822Z 
2022-05-18T03:56:03.7629916Z Generating XML reports...
2022-05-18T03:56:03.7662988Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220518035602.xml
2022-05-18T03:56:04.5069353Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5kvp86vk
2022-05-18T03:56:04.5069936Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5kvp86vk/_remote_module_non_scriptable.py
2022-05-18T03:56:04.7613278Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T03:56:04.7622578Z 
2022-05-18T03:56:04.7622675Z Running tests...
2022-05-18T03:56:04.7623413Z ----------------------------------------------------------------------
2022-05-18T03:56:05.0722465Z   test_call_python_function_remotely_from_script_not_supported (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27384
2022-05-18T03:56:05.0745612Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27385
2022-05-18T03:56:05.0769739Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27386
2022-05-18T03:56:05.0793888Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 27387
2022-05-18T03:56:05.7089710Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7ulif6ys
2022-05-18T03:56:05.7090499Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7ulif6ys/_remote_module_non_scriptable.py
2022-05-18T03:56:05.7213482Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9r36l0_8
2022-05-18T03:56:05.7214709Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9r36l0_8/_remote_module_non_scriptable.py
2022-05-18T03:56:05.7220935Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy_4n5pxw
2022-05-18T03:56:05.7223505Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy_4n5pxw/_remote_module_non_scriptable.py
2022-05-18T03:56:05.7378101Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo5vp9zey
2022-05-18T03:56:05.7379251Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo5vp9zey/_remote_module_non_scriptable.py
2022-05-18T03:56:05.9570624Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:56:05.9681376Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:56:05.9704332Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:56:05.9842457Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:56:06.4834127Z ok (1.721s)
2022-05-18T03:56:06.4834306Z 
2022-05-18T03:56:06.4834734Z ----------------------------------------------------------------------
2022-05-18T03:56:06.4835038Z Ran 1 test in 1.721s
2022-05-18T03:56:06.4835154Z 
2022-05-18T03:56:06.4835215Z OK
2022-05-18T03:56:06.4836339Z 
2022-05-18T03:56:06.4836839Z Generating XML reports...
2022-05-18T03:56:06.4870314Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220518035604.xml
2022-05-18T03:56:07.2541033Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz0xndehj
2022-05-18T03:56:07.2541589Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz0xndehj/_remote_module_non_scriptable.py
2022-05-18T03:56:07.5109529Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T03:56:07.5119166Z 
2022-05-18T03:56:07.5119259Z Running tests...
2022-05-18T03:56:07.5119836Z ----------------------------------------------------------------------
2022-05-18T03:56:07.8305612Z   test_call_rpc_with_profiling (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27607
2022-05-18T03:56:07.8328633Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27608
2022-05-18T03:56:07.8351903Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27609
2022-05-18T03:56:07.8376459Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 27610
2022-05-18T03:56:08.4592699Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxnqg19cr
2022-05-18T03:56:08.4593804Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxnqg19cr/_remote_module_non_scriptable.py
2022-05-18T03:56:08.4682133Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa4a8r428
2022-05-18T03:56:08.4683146Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa4a8r428/_remote_module_non_scriptable.py
2022-05-18T03:56:08.4834634Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy0tly7by
2022-05-18T03:56:08.4835654Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy0tly7by/_remote_module_non_scriptable.py
2022-05-18T03:56:08.4971533Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp15hywp5
2022-05-18T03:56:08.4972512Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp15hywp5/_remote_module_non_scriptable.py
2022-05-18T03:56:08.7115683Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:56:08.7154981Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:56:08.7317735Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:56:08.7464801Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:56:09.2417800Z ok (1.730s)
2022-05-18T03:56:09.2418011Z 
2022-05-18T03:56:09.2418395Z ----------------------------------------------------------------------
2022-05-18T03:56:09.2418649Z Ran 1 test in 1.730s
2022-05-18T03:56:09.2418768Z 
2022-05-18T03:56:09.2418816Z OK
2022-05-18T03:56:09.2418910Z 
2022-05-18T03:56:09.2419023Z Generating XML reports...
2022-05-18T03:56:09.2453802Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220518035607.xml
2022-05-18T03:56:10.0382034Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf5vzy6ej
2022-05-18T03:56:10.0383154Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf5vzy6ej/_remote_module_non_scriptable.py
2022-05-18T03:56:10.2913972Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T03:56:10.2923745Z 
2022-05-18T03:56:10.2924200Z Running tests...
2022-05-18T03:56:10.2924611Z ----------------------------------------------------------------------
2022-05-18T03:56:10.6054451Z   test_call_script_function_that_not_exists_remotely_from_script (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27830
2022-05-18T03:56:10.6077723Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27831
2022-05-18T03:56:10.6100814Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27832
2022-05-18T03:56:10.6125071Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 27833
2022-05-18T03:56:11.2305665Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpicmvylu6
2022-05-18T03:56:11.2306443Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpicmvylu6/_remote_module_non_scriptable.py
2022-05-18T03:56:11.2399145Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6foe1mth
2022-05-18T03:56:11.2400080Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6foe1mth/_remote_module_non_scriptable.py
2022-05-18T03:56:11.2412568Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf8uhbnqc
2022-05-18T03:56:11.2414438Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf8uhbnqc/_remote_module_non_scriptable.py
2022-05-18T03:56:11.2453028Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptqcgt811
2022-05-18T03:56:11.2454802Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptqcgt811/_remote_module_non_scriptable.py
2022-05-18T03:56:11.4771717Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:56:11.4854297Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:56:11.4919839Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:56:11.4932997Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:56:12.0165958Z ok (1.724s)
2022-05-18T03:56:12.0166186Z 
2022-05-18T03:56:12.0166589Z ----------------------------------------------------------------------
2022-05-18T03:56:12.0166843Z Ran 1 test in 1.724s
2022-05-18T03:56:12.0166997Z 
2022-05-18T03:56:12.0167068Z OK
2022-05-18T03:56:12.0167175Z 
2022-05-18T03:56:12.0167267Z Generating XML reports...
2022-05-18T03:56:12.0201188Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220518035610.xml
2022-05-18T03:56:12.7877757Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd7exq4xq
2022-05-18T03:56:12.7878529Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd7exq4xq/_remote_module_non_scriptable.py
2022-05-18T03:56:13.0395034Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T03:56:13.0404908Z 
2022-05-18T03:56:13.0405384Z Running tests...
2022-05-18T03:56:13.0405785Z ----------------------------------------------------------------------
2022-05-18T03:56:13.3533591Z   test_call_script_function_that_raises_remotely_from_script (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28053
2022-05-18T03:56:13.3556477Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28054
2022-05-18T03:56:13.3579946Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28055
2022-05-18T03:56:13.3603880Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 28056
2022-05-18T03:56:14.0365990Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdtbt_yu8
2022-05-18T03:56:14.0366775Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdtbt_yu8/_remote_module_non_scriptable.py
2022-05-18T03:56:14.0624120Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpns8lt4o3
2022-05-18T03:56:14.0624882Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpns8lt4o3/_remote_module_non_scriptable.py
2022-05-18T03:56:14.0986089Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4y6xwouo
2022-05-18T03:56:14.0987094Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4y6xwouo/_remote_module_non_scriptable.py
2022-05-18T03:56:14.0994346Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptobnmmrg
2022-05-18T03:56:14.0996385Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptobnmmrg/_remote_module_non_scriptable.py
2022-05-18T03:56:14.2868984Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:56:14.3120487Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:56:14.3450589Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:56:14.3467186Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:56:14.9648399Z ok (1.924s)
2022-05-18T03:56:14.9648601Z 
2022-05-18T03:56:14.9648959Z ----------------------------------------------------------------------
2022-05-18T03:56:14.9649214Z Ran 1 test in 1.924s
2022-05-18T03:56:14.9649330Z 
2022-05-18T03:56:14.9649408Z OK
2022-05-18T03:56:14.9649501Z 
2022-05-18T03:56:14.9649597Z Generating XML reports...
2022-05-18T03:56:14.9683892Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220518035613.xml
2022-05-18T03:56:15.7391140Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpoxulvh89
2022-05-18T03:56:15.7392050Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpoxulvh89/_remote_module_non_scriptable.py
2022-05-18T03:56:15.9936995Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T03:56:15.9946770Z 
2022-05-18T03:56:15.9946862Z Running tests...
2022-05-18T03:56:15.9947315Z ----------------------------------------------------------------------
2022-05-18T03:56:16.3202112Z   test_callback_chain (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28276
2022-05-18T03:56:16.3225515Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28277
2022-05-18T03:56:16.3248690Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28278
2022-05-18T03:56:16.3273010Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 28279
2022-05-18T03:56:16.9301975Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpah4ugb42
2022-05-18T03:56:16.9303113Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpah4ugb42/_remote_module_non_scriptable.py
2022-05-18T03:56:16.9433158Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpabdpcthn
2022-05-18T03:56:16.9434298Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpabdpcthn/_remote_module_non_scriptable.py
2022-05-18T03:56:16.9691528Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwn315i8v
2022-05-18T03:56:16.9692794Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwn315i8v/_remote_module_non_scriptable.py
2022-05-18T03:56:16.9806410Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppjuwitzc
2022-05-18T03:56:16.9807756Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppjuwitzc/_remote_module_non_scriptable.py
2022-05-18T03:56:17.1777822Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:56:17.1894348Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:56:17.2182297Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:56:17.2299683Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:56:17.7313036Z ok (1.736s)
2022-05-18T03:56:17.7313299Z 
2022-05-18T03:56:17.7313794Z ----------------------------------------------------------------------
2022-05-18T03:56:17.7314237Z Ran 1 test in 1.737s
2022-05-18T03:56:17.7314442Z 
2022-05-18T03:56:17.7314886Z OK
2022-05-18T03:56:17.7315064Z 
2022-05-18T03:56:17.7315187Z Generating XML reports...
2022-05-18T03:56:17.7349661Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220518035615.xml
2022-05-18T03:56:18.5117462Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk222uy46
2022-05-18T03:56:18.5118412Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk222uy46/_remote_module_non_scriptable.py
2022-05-18T03:56:18.7675053Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T03:56:18.7685119Z 
2022-05-18T03:56:18.7685411Z Running tests...
2022-05-18T03:56:18.7686078Z ----------------------------------------------------------------------
2022-05-18T03:56:19.0865316Z   test_callback_simple (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28495
2022-05-18T03:56:19.0888319Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28496
2022-05-18T03:56:19.0911274Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28497
2022-05-18T03:56:19.0936440Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 28498
2022-05-18T03:56:19.6863124Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgpdrv2pg
2022-05-18T03:56:19.6863876Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgpdrv2pg/_remote_module_non_scriptable.py
2022-05-18T03:56:19.6991351Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp7nv0c16
2022-05-18T03:56:19.6993311Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp7nv0c16/_remote_module_non_scriptable.py
2022-05-18T03:56:19.7134162Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi15dv8w4
2022-05-18T03:56:19.7135425Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi15dv8w4/_remote_module_non_scriptable.py
2022-05-18T03:56:19.7217441Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpog_6f68c
2022-05-18T03:56:19.7218727Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpog_6f68c/_remote_module_non_scriptable.py
2022-05-18T03:56:19.9333083Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:56:19.9450062Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:56:19.9614318Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:56:19.9705117Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:56:20.4976960Z ok (1.729s)
2022-05-18T03:56:20.4977115Z 
2022-05-18T03:56:20.4977418Z ----------------------------------------------------------------------
2022-05-18T03:56:20.4977740Z Ran 1 test in 1.729s
2022-05-18T03:56:20.4977857Z 
2022-05-18T03:56:20.4977931Z OK
2022-05-18T03:56:20.4978023Z 
2022-05-18T03:56:20.4978116Z Generating XML reports...
2022-05-18T03:56:20.5011755Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220518035618.xml
2022-05-18T03:56:21.2617434Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn5xzusne
2022-05-18T03:56:21.2618695Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn5xzusne/_remote_module_non_scriptable.py
2022-05-18T03:56:21.5148202Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T03:56:21.5158042Z 
2022-05-18T03:56:21.5158444Z Running tests...
2022-05-18T03:56:21.5158871Z ----------------------------------------------------------------------
2022-05-18T03:56:21.8275510Z   test_callback_with_exception (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28730
2022-05-18T03:56:21.8298891Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28731
2022-05-18T03:56:21.8321992Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28732
2022-05-18T03:56:21.8346364Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 28733
2022-05-18T03:56:22.4577516Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5u8v24i4
2022-05-18T03:56:22.4578250Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5u8v24i4/_remote_module_non_scriptable.py
2022-05-18T03:56:22.4601411Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo8qay43x
2022-05-18T03:56:22.4603325Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo8qay43x/_remote_module_non_scriptable.py
2022-05-18T03:56:22.4645927Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqqdbyo36
2022-05-18T03:56:22.4647835Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqqdbyo36/_remote_module_non_scriptable.py
2022-05-18T03:56:22.4669183Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsjcudbig
2022-05-18T03:56:22.4671837Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsjcudbig/_remote_module_non_scriptable.py
2022-05-18T03:56:22.7054145Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:56:22.7060505Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:56:22.7134124Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:56:22.7151246Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:56:23.3388554Z ok (1.823s)
2022-05-18T03:56:23.3388784Z 
2022-05-18T03:56:23.3389261Z ----------------------------------------------------------------------
2022-05-18T03:56:23.3389657Z Ran 1 test in 1.823s
2022-05-18T03:56:23.3389871Z 
2022-05-18T03:56:23.3390000Z OK
2022-05-18T03:56:23.3390170Z 
2022-05-18T03:56:23.3390332Z Generating XML reports...
2022-05-18T03:56:23.3425473Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220518035621.xml
2022-05-18T03:56:24.1045782Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcaq4h0k3
2022-05-18T03:56:24.1046266Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcaq4h0k3/_remote_module_non_scriptable.py
2022-05-18T03:56:24.3572575Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T03:56:24.3582011Z 
2022-05-18T03:56:24.3582109Z Running tests...
2022-05-18T03:56:24.3582644Z ----------------------------------------------------------------------
2022-05-18T03:56:24.6697526Z   test_create_local_script_class_rref_in_py (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28965
2022-05-18T03:56:24.6720236Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28966
2022-05-18T03:56:24.6743719Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28967
2022-05-18T03:56:24.6767472Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 28968
2022-05-18T03:56:25.3005525Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_2_q3xoa
2022-05-18T03:56:25.3006788Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_2_q3xoa/_remote_module_non_scriptable.py
2022-05-18T03:56:25.3334194Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplz2x21ak
2022-05-18T03:56:25.3335724Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplz2x21ak/_remote_module_non_scriptable.py
2022-05-18T03:56:25.3443645Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd3epa1_h
2022-05-18T03:56:25.3444434Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd3epa1_h/_remote_module_non_scriptable.py
2022-05-18T03:56:25.3610738Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp2n3c41i
2022-05-18T03:56:25.3611724Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp2n3c41i/_remote_module_non_scriptable.py
2022-05-18T03:56:25.5507260Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:56:25.5794342Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:56:25.5917508Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:56:25.6087927Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:56:26.0809215Z ok (1.722s)
2022-05-18T03:56:26.0809433Z 
2022-05-18T03:56:26.0809963Z ----------------------------------------------------------------------
2022-05-18T03:56:26.0810324Z Ran 1 test in 1.723s
2022-05-18T03:56:26.0810459Z 
2022-05-18T03:56:26.0810523Z OK
2022-05-18T03:56:26.0810617Z 
2022-05-18T03:56:26.0810712Z Generating XML reports...
2022-05-18T03:56:26.0844502Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220518035624.xml
2022-05-18T03:56:26.8457440Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjuewuvez
2022-05-18T03:56:26.8457964Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjuewuvez/_remote_module_non_scriptable.py
2022-05-18T03:56:27.0990264Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T03:56:27.0999481Z 
2022-05-18T03:56:27.0999591Z Running tests...
2022-05-18T03:56:27.1000171Z ----------------------------------------------------------------------
2022-05-18T03:56:27.4137702Z   test_create_local_script_module_rref_in_py (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29184
2022-05-18T03:56:27.4160829Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29185
2022-05-18T03:56:27.4184956Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29186
2022-05-18T03:56:27.4208638Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 29187
2022-05-18T03:56:28.0401872Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnlsi8zuv
2022-05-18T03:56:28.0402943Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnlsi8zuv/_remote_module_non_scriptable.py
2022-05-18T03:56:28.0422246Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp31h22tp_
2022-05-18T03:56:28.0424261Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp31h22tp_/_remote_module_non_scriptable.py
2022-05-18T03:56:28.0955573Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwaexfsiw
2022-05-18T03:56:28.0956414Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwaexfsiw/_remote_module_non_scriptable.py
2022-05-18T03:56:28.1257502Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx1sxdqqv
2022-05-18T03:56:28.1259088Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx1sxdqqv/_remote_module_non_scriptable.py
2022-05-18T03:56:28.2887027Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:56:28.2911035Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:56:28.3443450Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:56:28.3730622Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:56:28.8249185Z ok (1.725s)
2022-05-18T03:56:28.8249396Z 
2022-05-18T03:56:28.8249707Z ----------------------------------------------------------------------
2022-05-18T03:56:28.8249959Z Ran 1 test in 1.725s
2022-05-18T03:56:28.8250314Z 
2022-05-18T03:56:28.8250363Z OK
2022-05-18T03:56:28.8250456Z 
2022-05-18T03:56:28.8250549Z Generating XML reports...
2022-05-18T03:56:28.8284215Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220518035627.xml
2022-05-18T03:56:29.6020982Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe31866z4
2022-05-18T03:56:29.6021710Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe31866z4/_remote_module_non_scriptable.py
2022-05-18T03:56:29.8543914Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T03:56:29.8554473Z 
2022-05-18T03:56:29.8554769Z Running tests...
2022-05-18T03:56:29.8555394Z ----------------------------------------------------------------------
2022-05-18T03:56:30.1710500Z   test_create_script_module_on_remote (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29403
2022-05-18T03:56:30.1732723Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29404
2022-05-18T03:56:30.1755681Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29405
2022-05-18T03:56:30.1779975Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 29406
2022-05-18T03:56:30.8062825Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp98m02dx2
2022-05-18T03:56:30.8064373Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp98m02dx2/_remote_module_non_scriptable.py
2022-05-18T03:56:30.8245702Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpovyr_v_e
2022-05-18T03:56:30.8247118Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpovyr_v_e/_remote_module_non_scriptable.py
2022-05-18T03:56:30.8311081Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbtm_z1z4
2022-05-18T03:56:30.8312633Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbtm_z1z4/_remote_module_non_scriptable.py
2022-05-18T03:56:30.8455029Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmply25ub14
2022-05-18T03:56:30.8456565Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmply25ub14/_remote_module_non_scriptable.py
2022-05-18T03:56:31.0551243Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:56:31.0706899Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:56:31.0803876Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:56:31.0929648Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:56:31.5819873Z ok (1.726s)
2022-05-18T03:56:31.5820232Z 
2022-05-18T03:56:31.5820558Z ----------------------------------------------------------------------
2022-05-18T03:56:31.5820796Z Ran 1 test in 1.726s
2022-05-18T03:56:31.5820937Z 
2022-05-18T03:56:31.5821000Z OK
2022-05-18T03:56:31.5821094Z 
2022-05-18T03:56:31.5821193Z Generating XML reports...
2022-05-18T03:56:31.5855786Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220518035629.xml
2022-05-18T03:56:32.3545935Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_0alwvyk
2022-05-18T03:56:32.3546715Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_0alwvyk/_remote_module_non_scriptable.py
2022-05-18T03:56:32.6074717Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T03:56:32.6084866Z 
2022-05-18T03:56:32.6085135Z Running tests...
2022-05-18T03:56:32.6085771Z ----------------------------------------------------------------------
2022-05-18T03:56:32.9263721Z   test_future_passed_between_python_and_jit (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29622
2022-05-18T03:56:32.9285727Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29623
2022-05-18T03:56:32.9309175Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29624
2022-05-18T03:56:32.9333346Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 29625
2022-05-18T03:56:33.5813999Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpalws61wy
2022-05-18T03:56:33.5814776Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpalws61wy/_remote_module_non_scriptable.py
2022-05-18T03:56:33.6244281Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpimksulh1
2022-05-18T03:56:33.6245040Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpimksulh1/_remote_module_non_scriptable.py
2022-05-18T03:56:33.6359948Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqps41cq5
2022-05-18T03:56:33.6361223Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqps41cq5/_remote_module_non_scriptable.py
2022-05-18T03:56:33.6542373Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp85p0yds5
2022-05-18T03:56:33.6543566Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp85p0yds5/_remote_module_non_scriptable.py
2022-05-18T03:56:33.8310159Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:56:33.8726204Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:56:33.8848481Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:56:33.9000542Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:56:34.5376968Z ok (1.929s)
2022-05-18T03:56:34.5377176Z 
2022-05-18T03:56:34.5377599Z ----------------------------------------------------------------------
2022-05-18T03:56:34.5377874Z Ran 1 test in 1.929s
2022-05-18T03:56:34.5377992Z 
2022-05-18T03:56:34.5378060Z OK
2022-05-18T03:56:34.5378139Z 
2022-05-18T03:56:34.5378233Z Generating XML reports...
2022-05-18T03:56:34.5412188Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220518035632.xml
2022-05-18T03:56:35.2948366Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1tvnzdg9
2022-05-18T03:56:35.2949115Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1tvnzdg9/_remote_module_non_scriptable.py
2022-05-18T03:56:35.5480088Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T03:56:35.5489539Z 
2022-05-18T03:56:35.5489635Z Running tests...
2022-05-18T03:56:35.5490082Z ----------------------------------------------------------------------
2022-05-18T03:56:35.8634274Z   test_future_python_annotation (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29841
2022-05-18T03:56:35.8657580Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29842
2022-05-18T03:56:35.8680856Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29843
2022-05-18T03:56:35.8705586Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 29844
2022-05-18T03:56:36.5449530Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb72yqnja
2022-05-18T03:56:36.5450691Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb72yqnja/_remote_module_non_scriptable.py
2022-05-18T03:56:36.5790632Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc4zgsdz4
2022-05-18T03:56:36.5791570Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc4zgsdz4/_remote_module_non_scriptable.py
2022-05-18T03:56:36.5855351Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf5qfxolo
2022-05-18T03:56:36.5856722Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf5qfxolo/_remote_module_non_scriptable.py
2022-05-18T03:56:36.5928685Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp32k67mum
2022-05-18T03:56:36.5930510Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp32k67mum/_remote_module_non_scriptable.py
2022-05-18T03:56:36.7932710Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:56:36.8256834Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:56:36.8321803Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:56:36.8393029Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:56:37.3748082Z ok (1.826s)
2022-05-18T03:56:37.3748289Z 
2022-05-18T03:56:37.3748650Z ----------------------------------------------------------------------
2022-05-18T03:56:37.3748926Z Ran 1 test in 1.826s
2022-05-18T03:56:37.3749042Z 
2022-05-18T03:56:37.3749112Z OK
2022-05-18T03:56:37.3749259Z 
2022-05-18T03:56:37.3749344Z Generating XML reports...
2022-05-18T03:56:37.3782784Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220518035635.xml
2022-05-18T03:56:38.1564461Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuij3supu
2022-05-18T03:56:38.1565178Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuij3supu/_remote_module_non_scriptable.py
2022-05-18T03:56:38.4079059Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T03:56:38.4088570Z 
2022-05-18T03:56:38.4088712Z Running tests...
2022-05-18T03:56:38.4089260Z ----------------------------------------------------------------------
2022-05-18T03:56:38.7231444Z   test_kwargs_not_passed (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30064
2022-05-18T03:56:38.7254606Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30065
2022-05-18T03:56:38.7278012Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30066
2022-05-18T03:56:38.7301983Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 30067
2022-05-18T03:56:39.4081364Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4n958kaj
2022-05-18T03:56:39.4082165Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4n958kaj/_remote_module_non_scriptable.py
2022-05-18T03:56:39.4387814Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5u3cie08
2022-05-18T03:56:39.4388518Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7ngabcy4
2022-05-18T03:56:39.4389205Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5u3cie08/_remote_module_non_scriptable.py
2022-05-18T03:56:39.4389890Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7ngabcy4/_remote_module_non_scriptable.py
2022-05-18T03:56:39.4488971Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps0izeldq
2022-05-18T03:56:39.4490634Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps0izeldq/_remote_module_non_scriptable.py
2022-05-18T03:56:39.6565207Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:56:39.6846720Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:56:39.6864871Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:56:39.6954470Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:56:40.2345430Z ok (1.825s)
2022-05-18T03:56:40.2345774Z 
2022-05-18T03:56:40.2346122Z ----------------------------------------------------------------------
2022-05-18T03:56:40.2346630Z Ran 1 test in 1.826s
2022-05-18T03:56:40.2346746Z 
2022-05-18T03:56:40.2346809Z OK
2022-05-18T03:56:40.2346903Z 
2022-05-18T03:56:40.2346989Z Generating XML reports...
2022-05-18T03:56:40.2379674Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220518035638.xml
2022-05-18T03:56:41.0049057Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjhb1_o3k
2022-05-18T03:56:41.0049759Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjhb1_o3k/_remote_module_non_scriptable.py
2022-05-18T03:56:41.2576469Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T03:56:41.2585999Z 
2022-05-18T03:56:41.2586299Z Running tests...
2022-05-18T03:56:41.2586984Z ----------------------------------------------------------------------
2022-05-18T03:56:41.5707645Z   test_less_than_needed_args_are_specified (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30287
2022-05-18T03:56:41.5730637Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30288
2022-05-18T03:56:41.5754030Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30289
2022-05-18T03:56:41.5778013Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 30290
2022-05-18T03:56:42.2358602Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxf4642ky
2022-05-18T03:56:42.2359510Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxf4642ky/_remote_module_non_scriptable.py
2022-05-18T03:56:42.2934198Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4yuc6jwz
2022-05-18T03:56:42.2935002Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4yuc6jwz/_remote_module_non_scriptable.py
2022-05-18T03:56:42.3126307Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4x_ht74c
2022-05-18T03:56:42.3127118Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4x_ht74c/_remote_module_non_scriptable.py
2022-05-18T03:56:42.3371670Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp07k1t86w
2022-05-18T03:56:42.3372960Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp07k1t86w/_remote_module_non_scriptable.py
2022-05-18T03:56:42.4872093Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:56:42.5441630Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:56:42.5609903Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:56:42.5827298Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:56:43.0819627Z ok (1.823s)
2022-05-18T03:56:43.0819871Z 
2022-05-18T03:56:43.0820389Z ----------------------------------------------------------------------
2022-05-18T03:56:43.0820770Z Ran 1 test in 1.823s
2022-05-18T03:56:43.0820892Z 
2022-05-18T03:56:43.0820941Z OK
2022-05-18T03:56:43.0821033Z 
2022-05-18T03:56:43.0821124Z Generating XML reports...
2022-05-18T03:56:43.0854773Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220518035641.xml
2022-05-18T03:56:43.8703125Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi9g32u33
2022-05-18T03:56:43.8704028Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi9g32u33/_remote_module_non_scriptable.py
2022-05-18T03:56:44.1217576Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T03:56:44.1227761Z 
2022-05-18T03:56:44.1228303Z Running tests...
2022-05-18T03:56:44.1228707Z ----------------------------------------------------------------------
2022-05-18T03:56:44.4350344Z   test_load_script_module_with_pickled_rref (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30506
2022-05-18T03:56:44.4372149Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30507
2022-05-18T03:56:44.4395741Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30508
2022-05-18T03:56:44.4420471Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 30509
2022-05-18T03:56:45.0562537Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6v7_biie
2022-05-18T03:56:45.0563320Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6v7_biie/_remote_module_non_scriptable.py
2022-05-18T03:56:45.1068669Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeqg2otl0
2022-05-18T03:56:45.1069662Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeqg2otl0/_remote_module_non_scriptable.py
2022-05-18T03:56:45.1092023Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg97iejzv
2022-05-18T03:56:45.1093816Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg97iejzv/_remote_module_non_scriptable.py
2022-05-18T03:56:45.1289651Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsqe9xsh6
2022-05-18T03:56:45.1290615Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsqe9xsh6/_remote_module_non_scriptable.py
2022-05-18T03:56:45.3041297Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:56:45.3564302Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:56:45.3576244Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:56:45.3770715Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:56:45.9462192Z ok (1.823s)
2022-05-18T03:56:45.9462424Z 
2022-05-18T03:56:45.9463092Z ----------------------------------------------------------------------
2022-05-18T03:56:45.9463505Z Ran 1 test in 1.823s
2022-05-18T03:56:45.9463680Z 
2022-05-18T03:56:45.9463777Z OK
2022-05-18T03:56:45.9463925Z 
2022-05-18T03:56:45.9464053Z Generating XML reports...
2022-05-18T03:56:45.9497980Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220518035644.xml
2022-05-18T03:56:46.7113671Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps49kr7fz
2022-05-18T03:56:46.7114652Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps49kr7fz/_remote_module_non_scriptable.py
2022-05-18T03:56:46.9632274Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T03:56:46.9641690Z 
2022-05-18T03:56:46.9641816Z Running tests...
2022-05-18T03:56:46.9642213Z ----------------------------------------------------------------------
2022-05-18T03:56:47.2801829Z   test_local_rref_local_value (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30725
2022-05-18T03:56:47.2827211Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30726
2022-05-18T03:56:47.2850756Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30727
2022-05-18T03:56:47.2875765Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 30728
2022-05-18T03:56:47.8623579Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3tmowm_1
2022-05-18T03:56:47.8624730Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3tmowm_1/_remote_module_non_scriptable.py
2022-05-18T03:56:47.8652480Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcxn5yyvr
2022-05-18T03:56:47.8654191Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcxn5yyvr/_remote_module_non_scriptable.py
2022-05-18T03:56:47.9050903Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpujqgkaur
2022-05-18T03:56:47.9051957Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpujqgkaur/_remote_module_non_scriptable.py
2022-05-18T03:56:47.9121963Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd8ulsxq9
2022-05-18T03:56:47.9123059Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd8ulsxq9/_remote_module_non_scriptable.py
2022-05-18T03:56:48.1084316Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:56:48.1118341Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:56:48.1528357Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:56:48.1583322Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:56:48.6916654Z ok (1.727s)
2022-05-18T03:56:48.6916845Z 
2022-05-18T03:56:48.6917249Z ----------------------------------------------------------------------
2022-05-18T03:56:48.6917593Z Ran 1 test in 1.727s
2022-05-18T03:56:48.6917715Z 
2022-05-18T03:56:48.6917777Z OK
2022-05-18T03:56:48.6917871Z 
2022-05-18T03:56:48.6917955Z Generating XML reports...
2022-05-18T03:56:48.6952495Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220518035646.xml
2022-05-18T03:56:49.4637204Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcnn2fxf2
2022-05-18T03:56:49.4637801Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcnn2fxf2/_remote_module_non_scriptable.py
2022-05-18T03:56:49.7159906Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T03:56:49.7169124Z 
2022-05-18T03:56:49.7169210Z Running tests...
2022-05-18T03:56:49.7169785Z ----------------------------------------------------------------------
2022-05-18T03:56:50.0322448Z   test_more_than_needed_args_are_specified (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30944
2022-05-18T03:56:50.0345695Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30945
2022-05-18T03:56:50.0368705Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30946
2022-05-18T03:56:50.0393711Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 30947
2022-05-18T03:56:50.6518151Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1qajq2tp
2022-05-18T03:56:50.6518999Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1qajq2tp/_remote_module_non_scriptable.py
2022-05-18T03:56:50.6644236Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm5w4m72r
2022-05-18T03:56:50.6645472Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm5w4m72r/_remote_module_non_scriptable.py
2022-05-18T03:56:50.6738030Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc3gsbjwd
2022-05-18T03:56:50.6739743Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc3gsbjwd/_remote_module_non_scriptable.py
2022-05-18T03:56:50.6866961Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpga5vroii
2022-05-18T03:56:50.6868173Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpga5vroii/_remote_module_non_scriptable.py
2022-05-18T03:56:50.9007790Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:56:50.9125557Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:56:50.9212476Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:56:50.9365491Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:56:51.3431637Z ok (1.626s)
2022-05-18T03:56:51.3431834Z 
2022-05-18T03:56:51.3432227Z ----------------------------------------------------------------------
2022-05-18T03:56:51.3432740Z Ran 1 test in 1.626s
2022-05-18T03:56:51.3432857Z 
2022-05-18T03:56:51.3432919Z OK
2022-05-18T03:56:51.3433010Z 
2022-05-18T03:56:51.3433167Z Generating XML reports...
2022-05-18T03:56:51.3466841Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220518035649.xml
2022-05-18T03:56:52.1111420Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplmrzzlvn
2022-05-18T03:56:52.1111901Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplmrzzlvn/_remote_module_non_scriptable.py
2022-05-18T03:56:52.3649342Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T03:56:52.3658912Z 
2022-05-18T03:56:52.3659044Z Running tests...
2022-05-18T03:56:52.3659798Z ----------------------------------------------------------------------
2022-05-18T03:56:52.6830423Z   test_my_script_module_with_rrefs (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31163
2022-05-18T03:56:52.6852534Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31164
2022-05-18T03:56:52.6876511Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31165
2022-05-18T03:56:52.6900535Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 31166
2022-05-18T03:56:53.3543363Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzjc4dzdz
2022-05-18T03:56:53.3544581Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzjc4dzdz/_remote_module_non_scriptable.py
2022-05-18T03:56:53.3972906Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqatr_8iu
2022-05-18T03:56:53.3974047Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqatr_8iu/_remote_module_non_scriptable.py
2022-05-18T03:56:53.4212244Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfpvz8ihs
2022-05-18T03:56:53.4213503Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfpvz8ihs/_remote_module_non_scriptable.py
2022-05-18T03:56:53.4433194Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7_mv6lr7
2022-05-18T03:56:53.4433941Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7_mv6lr7/_remote_module_non_scriptable.py
2022-05-18T03:56:53.6026510Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:56:53.6434922Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:56:53.6720226Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:56:53.6901590Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:56:54.1943558Z ok (1.828s)
2022-05-18T03:56:54.1943886Z 
2022-05-18T03:56:54.1944195Z ----------------------------------------------------------------------
2022-05-18T03:56:54.1944608Z Ran 1 test in 1.828s
2022-05-18T03:56:54.1945107Z 
2022-05-18T03:56:54.1945312Z OK
2022-05-18T03:56:54.1945564Z 
2022-05-18T03:56:54.1945743Z Generating XML reports...
2022-05-18T03:56:54.1978811Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220518035652.xml
2022-05-18T03:56:54.9583082Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7sdqm4c2
2022-05-18T03:56:54.9618456Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7sdqm4c2/_remote_module_non_scriptable.py
2022-05-18T03:56:55.2131805Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T03:56:55.2141104Z 
2022-05-18T03:56:55.2141295Z Running tests...
2022-05-18T03:56:55.2141713Z ----------------------------------------------------------------------
2022-05-18T03:56:55.5327529Z   test_no_kwargs_are_populated_by_defaults (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31382
2022-05-18T03:56:55.5350596Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31383
2022-05-18T03:56:55.5374273Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31384
2022-05-18T03:56:55.5398741Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 31385
2022-05-18T03:56:56.1509896Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5a0mzkkc
2022-05-18T03:56:56.1510696Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5a0mzkkc/_remote_module_non_scriptable.py
2022-05-18T03:56:56.1630206Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwi4wkacz
2022-05-18T03:56:56.1631803Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwi4wkacz/_remote_module_non_scriptable.py
2022-05-18T03:56:56.1814503Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkjycju91
2022-05-18T03:56:56.1815674Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkjycju91/_remote_module_non_scriptable.py
2022-05-18T03:56:56.1820082Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo5gi9ff0
2022-05-18T03:56:56.1820747Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo5gi9ff0/_remote_module_non_scriptable.py
2022-05-18T03:56:56.3983859Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:56:56.4087462Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:56:56.4290245Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:56:56.4295491Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:56:57.0438591Z ok (1.829s)
2022-05-18T03:56:57.0438806Z 
2022-05-18T03:56:57.0439514Z ----------------------------------------------------------------------
2022-05-18T03:56:57.0439823Z Ran 1 test in 1.830s
2022-05-18T03:56:57.0439949Z 
2022-05-18T03:56:57.0440012Z OK
2022-05-18T03:56:57.0440106Z 
2022-05-18T03:56:57.0440197Z Generating XML reports...
2022-05-18T03:56:57.0476198Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220518035655.xml
2022-05-18T03:56:57.8141511Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpizugfe5q
2022-05-18T03:56:57.8142338Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpizugfe5q/_remote_module_non_scriptable.py
2022-05-18T03:56:58.0664998Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T03:56:58.0674287Z 
2022-05-18T03:56:58.0674395Z Running tests...
2022-05-18T03:56:58.0674860Z ----------------------------------------------------------------------
2022-05-18T03:56:58.3864640Z   test_record_function_jit_end_callbacks_with_fork (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31605
2022-05-18T03:56:58.3887591Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31606
2022-05-18T03:56:58.3911496Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31607
2022-05-18T03:56:58.3936092Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 31608
2022-05-18T03:56:59.0204018Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4dsb7vsa
2022-05-18T03:56:59.0204776Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4dsb7vsa/_remote_module_non_scriptable.py
2022-05-18T03:56:59.0257783Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl7tig9q7
2022-05-18T03:56:59.0259097Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl7tig9q7/_remote_module_non_scriptable.py
2022-05-18T03:56:59.0276141Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0th3vdn1
2022-05-18T03:56:59.0277738Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0th3vdn1/_remote_module_non_scriptable.py
2022-05-18T03:56:59.0330552Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyteg3kds
2022-05-18T03:56:59.0331914Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyteg3kds/_remote_module_non_scriptable.py
2022-05-18T03:56:59.2700364Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:56:59.2737783Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:56:59.2772886Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:56:59.2795074Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:57:00.4984062Z ok (2.431s)
2022-05-18T03:57:00.4984261Z 
2022-05-18T03:57:00.4984801Z ----------------------------------------------------------------------
2022-05-18T03:57:00.4985246Z Ran 1 test in 2.431s
2022-05-18T03:57:00.4985410Z 
2022-05-18T03:57:00.4985471Z OK
2022-05-18T03:57:00.4985564Z 
2022-05-18T03:57:00.4985664Z Generating XML reports...
2022-05-18T03:57:00.5019900Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220518035658.xml
2022-05-18T03:57:01.3412294Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc17uuvb2
2022-05-18T03:57:01.3413583Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc17uuvb2/_remote_module_non_scriptable.py
2022-05-18T03:57:01.5937679Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T03:57:01.5947239Z 
2022-05-18T03:57:01.5947337Z Running tests...
2022-05-18T03:57:01.5948171Z ----------------------------------------------------------------------
2022-05-18T03:57:01.9114831Z   test_record_function_on_caller_rpc_async (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31660
2022-05-18T03:57:01.9139210Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31661
2022-05-18T03:57:01.9161937Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31662
2022-05-18T03:57:01.9186297Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 31663
2022-05-18T03:57:02.5684220Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuzj_jhe9
2022-05-18T03:57:02.5685001Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuzj_jhe9/_remote_module_non_scriptable.py
2022-05-18T03:57:02.5930148Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc4mjtl55
2022-05-18T03:57:02.5931473Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc4mjtl55/_remote_module_non_scriptable.py
2022-05-18T03:57:02.6219236Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt9n10vzv
2022-05-18T03:57:02.6220075Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt9n10vzv/_remote_module_non_scriptable.py
2022-05-18T03:57:02.6458044Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt5470jc0
2022-05-18T03:57:02.6458983Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt5470jc0/_remote_module_non_scriptable.py
2022-05-18T03:57:02.8186967Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:57:02.8394618Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:57:02.8698658Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:57:02.8924775Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:57:03.3298120Z ok (1.735s)
2022-05-18T03:57:03.3298684Z 
2022-05-18T03:57:03.3299164Z ----------------------------------------------------------------------
2022-05-18T03:57:03.3299530Z Ran 1 test in 1.735s
2022-05-18T03:57:03.3299709Z 
2022-05-18T03:57:03.3299811Z OK
2022-05-18T03:57:03.3300071Z 
2022-05-18T03:57:03.3300210Z Generating XML reports...
2022-05-18T03:57:03.3336973Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220518035701.xml
2022-05-18T03:57:04.1132301Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptym58f7a
2022-05-18T03:57:04.1133179Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptym58f7a/_remote_module_non_scriptable.py
2022-05-18T03:57:04.3664253Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T03:57:04.3674031Z 
2022-05-18T03:57:04.3674465Z Running tests...
2022-05-18T03:57:04.3675078Z ----------------------------------------------------------------------
2022-05-18T03:57:04.6822853Z   test_remote_script_module (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31883
2022-05-18T03:57:04.6846980Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31884
2022-05-18T03:57:04.6869711Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31885
2022-05-18T03:57:04.6894820Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 31886
2022-05-18T03:57:05.2786107Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsikcq9fx
2022-05-18T03:57:05.2787620Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsikcq9fx/_remote_module_non_scriptable.py
2022-05-18T03:57:05.3164641Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpameeubps
2022-05-18T03:57:05.3165746Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpameeubps/_remote_module_non_scriptable.py
2022-05-18T03:57:05.3214149Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp55o24uxg
2022-05-18T03:57:05.3215862Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp55o24uxg/_remote_module_non_scriptable.py
2022-05-18T03:57:05.3323940Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6cirg9mq
2022-05-18T03:57:05.3325327Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6cirg9mq/_remote_module_non_scriptable.py
2022-05-18T03:57:05.5291751Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:57:05.5654045Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:57:05.5702547Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:57:05.5784230Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:57:06.1935640Z ok (1.826s)
2022-05-18T03:57:06.1935899Z 
2022-05-18T03:57:06.1936224Z ----------------------------------------------------------------------
2022-05-18T03:57:06.1936466Z Ran 1 test in 1.826s
2022-05-18T03:57:06.1936583Z 
2022-05-18T03:57:06.1936646Z OK
2022-05-18T03:57:06.1936744Z 
2022-05-18T03:57:06.1936838Z Generating XML reports...
2022-05-18T03:57:06.1971803Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220518035704.xml
2022-05-18T03:57:06.9603565Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7ry9pkn5
2022-05-18T03:57:06.9604562Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7ry9pkn5/_remote_module_non_scriptable.py
2022-05-18T03:57:07.2139302Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T03:57:07.2149715Z 
2022-05-18T03:57:07.2149811Z Running tests...
2022-05-18T03:57:07.2150316Z ----------------------------------------------------------------------
2022-05-18T03:57:07.5377884Z   test_remote_script_throw (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32102
2022-05-18T03:57:07.5401093Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32103
2022-05-18T03:57:07.5424811Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 32104
2022-05-18T03:57:07.5449366Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 32105
2022-05-18T03:57:08.1518061Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuatcy360
2022-05-18T03:57:08.1519145Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuatcy360/_remote_module_non_scriptable.py
2022-05-18T03:57:08.1617764Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz88hwf43
2022-05-18T03:57:08.1619105Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz88hwf43/_remote_module_non_scriptable.py
2022-05-18T03:57:08.1838173Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps2is1sbl
2022-05-18T03:57:08.1839296Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps2is1sbl/_remote_module_non_scriptable.py
2022-05-18T03:57:08.1986905Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3kirmzh8
2022-05-18T03:57:08.1987645Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3kirmzh8/_remote_module_non_scriptable.py
2022-05-18T03:57:08.3997762Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:57:08.4107849Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:57:08.4352317Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:57:08.4468862Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:57:09.0491639Z ok (1.834s)
2022-05-18T03:57:09.0491846Z 
2022-05-18T03:57:09.0492643Z ----------------------------------------------------------------------
2022-05-18T03:57:09.0492975Z Ran 1 test in 1.834s
2022-05-18T03:57:09.0493095Z 
2022-05-18T03:57:09.0493158Z OK
2022-05-18T03:57:09.0493277Z 
2022-05-18T03:57:09.0493358Z Generating XML reports...
2022-05-18T03:57:09.0527090Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220518035707.xml
2022-05-18T03:57:09.8159988Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps3soln3s
2022-05-18T03:57:09.8161187Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps3soln3s/_remote_module_non_scriptable.py
2022-05-18T03:57:10.0682114Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T03:57:10.0691185Z 
2022-05-18T03:57:10.0691297Z Running tests...
2022-05-18T03:57:10.0691855Z ----------------------------------------------------------------------
2022-05-18T03:57:10.3876070Z   test_remote_script_udf (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32321
2022-05-18T03:57:10.3897544Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32322
2022-05-18T03:57:10.3921058Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 32323
2022-05-18T03:57:10.3944511Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 32324
2022-05-18T03:57:11.0249206Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxxs8q__a
2022-05-18T03:57:11.0250038Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxxs8q__a/_remote_module_non_scriptable.py
2022-05-18T03:57:11.0380710Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmi9rx2m5
2022-05-18T03:57:11.0381811Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmi9rx2m5/_remote_module_non_scriptable.py
2022-05-18T03:57:11.0440440Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpiyk89e_5
2022-05-18T03:57:11.0442062Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpiyk89e_5/_remote_module_non_scriptable.py
2022-05-18T03:57:11.0503372Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8trrzqkm
2022-05-18T03:57:11.0504578Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8trrzqkm/_remote_module_non_scriptable.py
2022-05-18T03:57:11.2740691Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:57:11.2851238Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:57:11.2891461Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:57:11.2963661Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:57:11.7987742Z ok (1.729s)
2022-05-18T03:57:11.7988003Z 
2022-05-18T03:57:11.7988521Z ----------------------------------------------------------------------
2022-05-18T03:57:11.7988813Z Ran 1 test in 1.730s
2022-05-18T03:57:11.7988930Z 
2022-05-18T03:57:11.7988994Z OK
2022-05-18T03:57:11.7989095Z 
2022-05-18T03:57:11.7989173Z Generating XML reports...
2022-05-18T03:57:11.8021906Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220518035710.xml
2022-05-18T03:57:12.5702469Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0z5vy86o
2022-05-18T03:57:12.5703456Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0z5vy86o/_remote_module_non_scriptable.py
2022-05-18T03:57:12.8230208Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T03:57:12.8240346Z 
2022-05-18T03:57:12.8240756Z Running tests...
2022-05-18T03:57:12.8241167Z ----------------------------------------------------------------------
2022-05-18T03:57:13.1382144Z   test_return_local_script_class_rref_in_py_and_use_in_script (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32556
2022-05-18T03:57:13.1405457Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32557
2022-05-18T03:57:13.1429483Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 32558
2022-05-18T03:57:13.1453603Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 32559
2022-05-18T03:57:13.7665526Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcxct_s72
2022-05-18T03:57:13.7666312Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcxct_s72/_remote_module_non_scriptable.py
2022-05-18T03:57:13.7748633Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa_9nrvmh
2022-05-18T03:57:13.7750101Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa_9nrvmh/_remote_module_non_scriptable.py
2022-05-18T03:57:13.7757573Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplk1cwo8b
2022-05-18T03:57:13.7759733Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplk1cwo8b/_remote_module_non_scriptable.py
2022-05-18T03:57:13.7842544Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprnr9vwno
2022-05-18T03:57:13.7843883Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprnr9vwno/_remote_module_non_scriptable.py
2022-05-18T03:57:14.0167648Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:57:14.0244905Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:57:14.0247492Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:57:14.0325733Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:57:14.6496166Z ok (1.825s)
2022-05-18T03:57:14.6496781Z 
2022-05-18T03:57:14.6497288Z ----------------------------------------------------------------------
2022-05-18T03:57:14.6497742Z Ran 1 test in 1.826s
2022-05-18T03:57:14.6497890Z 
2022-05-18T03:57:14.6498029Z OK
2022-05-18T03:57:14.6498125Z 
2022-05-18T03:57:14.6498219Z Generating XML reports...
2022-05-18T03:57:14.6531946Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220518035712.xml
2022-05-18T03:57:15.4298737Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyzu_h5oo
2022-05-18T03:57:15.4299748Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyzu_h5oo/_remote_module_non_scriptable.py
2022-05-18T03:57:15.6827835Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T03:57:15.6837601Z 
2022-05-18T03:57:15.6837728Z Running tests...
2022-05-18T03:57:15.6838281Z ----------------------------------------------------------------------
2022-05-18T03:57:15.9987957Z   test_return_local_script_module_rref_in_py_and_use_in_script (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 311
2022-05-18T03:57:16.0010505Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 312
2022-05-18T03:57:16.0033841Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 313
2022-05-18T03:57:16.0057931Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 314
2022-05-18T03:57:16.6351555Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdwaeyiq4
2022-05-18T03:57:16.6352394Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdwaeyiq4/_remote_module_non_scriptable.py
2022-05-18T03:57:16.6910436Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0b6myzs4
2022-05-18T03:57:16.6911254Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0b6myzs4/_remote_module_non_scriptable.py
2022-05-18T03:57:16.7516880Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpulm7pe3v
2022-05-18T03:57:16.7517676Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpulm7pe3v/_remote_module_non_scriptable.py
2022-05-18T03:57:16.7562706Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpszji37mc
2022-05-18T03:57:16.7563898Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpszji37mc/_remote_module_non_scriptable.py
2022-05-18T03:57:16.8854352Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:57:16.9404832Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:57:16.9992773Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:57:17.0038312Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:57:17.6101702Z ok (1.926s)
2022-05-18T03:57:17.6102177Z 
2022-05-18T03:57:17.6103245Z ----------------------------------------------------------------------
2022-05-18T03:57:17.6103764Z Ran 1 test in 1.926s
2022-05-18T03:57:17.6103984Z 
2022-05-18T03:57:17.6104059Z OK
2022-05-18T03:57:17.6104155Z 
2022-05-18T03:57:17.6104235Z Generating XML reports...
2022-05-18T03:57:17.6139254Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220518035715.xml
2022-05-18T03:57:18.3791740Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8tupaz31
2022-05-18T03:57:18.3792464Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8tupaz31/_remote_module_non_scriptable.py
2022-05-18T03:57:18.6314636Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T03:57:18.6324590Z 
2022-05-18T03:57:18.6324801Z Running tests...
2022-05-18T03:57:18.6325232Z ----------------------------------------------------------------------
2022-05-18T03:57:18.9518911Z   test_rpc_async_jit_profiled (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 534
2022-05-18T03:57:18.9541898Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 535
2022-05-18T03:57:18.9565662Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 536
2022-05-18T03:57:18.9590405Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 537
2022-05-18T03:57:19.5563232Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpst035grm
2022-05-18T03:57:19.5564050Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpst035grm/_remote_module_non_scriptable.py
2022-05-18T03:57:19.5791862Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg8jngpik
2022-05-18T03:57:19.5792618Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg8jngpik/_remote_module_non_scriptable.py
2022-05-18T03:57:19.5844005Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmr7ulsen
2022-05-18T03:57:19.5845314Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmr7ulsen/_remote_module_non_scriptable.py
2022-05-18T03:57:19.5860805Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmo915_s4
2022-05-18T03:57:19.5862805Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmo915_s4/_remote_module_non_scriptable.py
2022-05-18T03:57:19.8049967Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:57:19.8280586Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:57:19.8305730Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:57:19.8368643Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:57:20.3632108Z ok (1.730s)
2022-05-18T03:57:20.3632383Z 
2022-05-18T03:57:20.3632866Z ----------------------------------------------------------------------
2022-05-18T03:57:20.3633103Z Ran 1 test in 1.731s
2022-05-18T03:57:20.3633232Z 
2022-05-18T03:57:20.3633293Z OK
2022-05-18T03:57:20.3633384Z 
2022-05-18T03:57:20.3633477Z Generating XML reports...
2022-05-18T03:57:20.3666827Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220518035718.xml
2022-05-18T03:57:21.1436992Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpioj5_f45
2022-05-18T03:57:21.1437762Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpioj5_f45/_remote_module_non_scriptable.py
2022-05-18T03:57:21.3961099Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T03:57:21.3971101Z 
2022-05-18T03:57:21.3971211Z Running tests...
2022-05-18T03:57:21.3972225Z ----------------------------------------------------------------------
2022-05-18T03:57:21.7148943Z   test_rpc_torchscript_record_function (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 757
2022-05-18T03:57:21.7172010Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 758
2022-05-18T03:57:21.7195113Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 759
2022-05-18T03:57:21.7218386Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 760
2022-05-18T03:57:22.3369686Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp22ubrlrn
2022-05-18T03:57:22.3370465Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp22ubrlrn/_remote_module_non_scriptable.py
2022-05-18T03:57:22.3496382Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpehj582f8
2022-05-18T03:57:22.3497236Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpehj582f8/_remote_module_non_scriptable.py
2022-05-18T03:57:22.3547020Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfhb947sr
2022-05-18T03:57:22.3549146Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfhb947sr/_remote_module_non_scriptable.py
2022-05-18T03:57:22.3632689Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa7dhqk9b
2022-05-18T03:57:22.3634351Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa7dhqk9b/_remote_module_non_scriptable.py
2022-05-18T03:57:22.5842380Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:57:22.5997871Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:57:22.6031791Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:57:22.6124601Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:57:23.2261977Z ok (1.829s)
2022-05-18T03:57:23.2262208Z 
2022-05-18T03:57:23.2262733Z ----------------------------------------------------------------------
2022-05-18T03:57:23.2263294Z Ran 1 test in 1.829s
2022-05-18T03:57:23.2263425Z 
2022-05-18T03:57:23.2263487Z OK
2022-05-18T03:57:23.2263581Z 
2022-05-18T03:57:23.2263666Z Generating XML reports...
2022-05-18T03:57:23.2296112Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220518035721.xml
2022-05-18T03:57:24.0052460Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe4yug_tj
2022-05-18T03:57:24.0053620Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe4yug_tj/_remote_module_non_scriptable.py
2022-05-18T03:57:24.2568581Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T03:57:24.2578553Z 
2022-05-18T03:57:24.2578649Z Running tests...
2022-05-18T03:57:24.2579153Z ----------------------------------------------------------------------
2022-05-18T03:57:24.5731609Z   test_rref_as_arg_and_return (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 980
2022-05-18T03:57:24.5755148Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 981
2022-05-18T03:57:24.5778295Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 982
2022-05-18T03:57:24.5802601Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 983
2022-05-18T03:57:25.2507203Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu33pcbjy
2022-05-18T03:57:25.2507984Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu33pcbjy/_remote_module_non_scriptable.py
2022-05-18T03:57:25.2567655Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4b4gtn4r
2022-05-18T03:57:25.2568630Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4b4gtn4r/_remote_module_non_scriptable.py
2022-05-18T03:57:25.2716983Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvrn11yao
2022-05-18T03:57:25.2718081Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvrn11yao/_remote_module_non_scriptable.py
2022-05-18T03:57:25.3118586Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjkid48wm
2022-05-18T03:57:25.3119327Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjkid48wm/_remote_module_non_scriptable.py
2022-05-18T03:57:25.5002597Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:57:25.5068997Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:57:25.5213852Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:57:25.5609156Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:57:26.1846282Z ok (1.926s)
2022-05-18T03:57:26.1846539Z 
2022-05-18T03:57:26.1847084Z ----------------------------------------------------------------------
2022-05-18T03:57:26.1847506Z Ran 1 test in 1.927s
2022-05-18T03:57:26.1847868Z 
2022-05-18T03:57:26.1847931Z OK
2022-05-18T03:57:26.1848010Z 
2022-05-18T03:57:26.1848106Z Generating XML reports...
2022-05-18T03:57:26.1881719Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220518035724.xml
2022-05-18T03:57:26.9553846Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0mzu302n
2022-05-18T03:57:26.9554637Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0mzu302n/_remote_module_non_scriptable.py
2022-05-18T03:57:27.2086753Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T03:57:27.2096188Z 
2022-05-18T03:57:27.2096383Z Running tests...
2022-05-18T03:57:27.2096876Z ----------------------------------------------------------------------
2022-05-18T03:57:27.5279892Z   test_rref_is_owner (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1199
2022-05-18T03:57:27.5303275Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1200
2022-05-18T03:57:27.5326674Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1201
2022-05-18T03:57:27.5351078Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1202
2022-05-18T03:57:28.0990667Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc8012j2p
2022-05-18T03:57:28.0991882Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc8012j2p/_remote_module_non_scriptable.py
2022-05-18T03:57:28.1045849Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6vldzjvt
2022-05-18T03:57:28.1047431Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6vldzjvt/_remote_module_non_scriptable.py
2022-05-18T03:57:28.1372860Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp38atinxo
2022-05-18T03:57:28.1373615Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp38atinxo/_remote_module_non_scriptable.py
2022-05-18T03:57:28.1553457Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn361vh1x
2022-05-18T03:57:28.1554401Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn361vh1x/_remote_module_non_scriptable.py
2022-05-18T03:57:28.3505727Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:57:28.3537269Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:57:28.3866122Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:57:28.4049251Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:57:28.9399532Z ok (1.730s)
2022-05-18T03:57:28.9399814Z 
2022-05-18T03:57:28.9400325Z ----------------------------------------------------------------------
2022-05-18T03:57:28.9400567Z Ran 1 test in 1.730s
2022-05-18T03:57:28.9400701Z 
2022-05-18T03:57:28.9400766Z OK
2022-05-18T03:57:28.9400857Z 
2022-05-18T03:57:28.9400956Z Generating XML reports...
2022-05-18T03:57:28.9434381Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220518035727.xml
2022-05-18T03:57:29.7201811Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjt_q46pk
2022-05-18T03:57:29.7202338Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjt_q46pk/_remote_module_non_scriptable.py
2022-05-18T03:57:29.9721728Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T03:57:29.9731634Z 
2022-05-18T03:57:29.9731768Z Running tests...
2022-05-18T03:57:29.9732213Z ----------------------------------------------------------------------
2022-05-18T03:57:30.2905812Z   test_rref_jit_pickle_not_supported (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1418
2022-05-18T03:57:30.2930476Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1419
2022-05-18T03:57:30.2955500Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1420
2022-05-18T03:57:30.2981195Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1421
2022-05-18T03:57:30.8942080Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_zi5yjx9
2022-05-18T03:57:30.8943281Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_zi5yjx9/_remote_module_non_scriptable.py
2022-05-18T03:57:30.9011632Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphb0kck48
2022-05-18T03:57:30.9013035Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphb0kck48/_remote_module_non_scriptable.py
2022-05-18T03:57:30.9051607Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgbbgircs
2022-05-18T03:57:30.9053131Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgbbgircs/_remote_module_non_scriptable.py
2022-05-18T03:57:30.9188069Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx5u48lav
2022-05-18T03:57:30.9189607Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx5u48lav/_remote_module_non_scriptable.py
2022-05-18T03:57:31.1429031Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:57:31.1485456Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:57:31.1520300Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:57:31.1652969Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:57:31.7021955Z ok (1.729s)
2022-05-18T03:57:31.7022392Z 
2022-05-18T03:57:31.7023430Z ----------------------------------------------------------------------
2022-05-18T03:57:31.7023907Z Ran 1 test in 1.729s
2022-05-18T03:57:31.7024054Z 
2022-05-18T03:57:31.7024116Z OK
2022-05-18T03:57:31.7024210Z 
2022-05-18T03:57:31.7024291Z Generating XML reports...
2022-05-18T03:57:31.7057436Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220518035729.xml
2022-05-18T03:57:32.4731268Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_85gyxdm
2022-05-18T03:57:32.4731966Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_85gyxdm/_remote_module_non_scriptable.py
2022-05-18T03:57:32.7253313Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T03:57:32.7263724Z 
2022-05-18T03:57:32.7263862Z Running tests...
2022-05-18T03:57:32.7264441Z ----------------------------------------------------------------------
2022-05-18T03:57:33.0400010Z   test_rref_list_mutate (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1637
2022-05-18T03:57:33.0423114Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1638
2022-05-18T03:57:33.0445922Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1639
2022-05-18T03:57:33.0470211Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1640
2022-05-18T03:57:33.6889845Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpem7174h2
2022-05-18T03:57:33.6890602Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpem7174h2/_remote_module_non_scriptable.py
2022-05-18T03:57:33.6961529Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvxx7mrw9
2022-05-18T03:57:33.6962757Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvxx7mrw9/_remote_module_non_scriptable.py
2022-05-18T03:57:33.7085845Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc3omf07t
2022-05-18T03:57:33.7086764Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc3omf07t/_remote_module_non_scriptable.py
2022-05-18T03:57:33.7183359Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpofwzck_u
2022-05-18T03:57:33.7184813Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpofwzck_u/_remote_module_non_scriptable.py
2022-05-18T03:57:33.9387589Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:57:33.9446275Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:57:33.9569538Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:57:33.9669445Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:57:34.5512536Z ok (1.825s)
2022-05-18T03:57:34.5512771Z 
2022-05-18T03:57:34.5513192Z ----------------------------------------------------------------------
2022-05-18T03:57:34.5513495Z Ran 1 test in 1.825s
2022-05-18T03:57:34.5513632Z 
2022-05-18T03:57:34.5513695Z OK
2022-05-18T03:57:34.5513789Z 
2022-05-18T03:57:34.5513887Z Generating XML reports...
2022-05-18T03:57:34.5547048Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220518035732.xml
2022-05-18T03:57:35.3205525Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphe014sd_
2022-05-18T03:57:35.3205992Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphe014sd_/_remote_module_non_scriptable.py
2022-05-18T03:57:35.5755325Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T03:57:35.5765101Z 
2022-05-18T03:57:35.5765243Z Running tests...
2022-05-18T03:57:35.5765660Z ----------------------------------------------------------------------
2022-05-18T03:57:35.8912432Z   test_rref_local_value (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1856
2022-05-18T03:57:35.8935549Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1857
2022-05-18T03:57:35.8958197Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1858
2022-05-18T03:57:35.8982384Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1859
2022-05-18T03:57:36.5127243Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo4i9x8sd
2022-05-18T03:57:36.5127987Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo4i9x8sd/_remote_module_non_scriptable.py
2022-05-18T03:57:36.5210987Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphck0j5c7
2022-05-18T03:57:36.5212297Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphck0j5c7/_remote_module_non_scriptable.py
2022-05-18T03:57:36.5777009Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5cwztms1
2022-05-18T03:57:36.5778795Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5cwztms1/_remote_module_non_scriptable.py
2022-05-18T03:57:36.5963403Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpstogcc5e
2022-05-18T03:57:36.5965152Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpstogcc5e/_remote_module_non_scriptable.py
2022-05-18T03:57:36.7612973Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:57:36.7685159Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:57:36.8592672Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:57:36.8705085Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:57:37.5026928Z ok (1.926s)
2022-05-18T03:57:37.5027142Z 
2022-05-18T03:57:37.5027483Z ----------------------------------------------------------------------
2022-05-18T03:57:37.5027756Z Ran 1 test in 1.926s
2022-05-18T03:57:37.5028114Z 
2022-05-18T03:57:37.5028181Z OK
2022-05-18T03:57:37.5028274Z 
2022-05-18T03:57:37.5028368Z Generating XML reports...
2022-05-18T03:57:37.5061578Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220518035735.xml
2022-05-18T03:57:38.2736588Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg8gakypq
2022-05-18T03:57:38.2737099Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg8gakypq/_remote_module_non_scriptable.py
2022-05-18T03:57:38.5263109Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T03:57:38.5272520Z 
2022-05-18T03:57:38.5272631Z Running tests...
2022-05-18T03:57:38.5273206Z ----------------------------------------------------------------------
2022-05-18T03:57:38.8466167Z   test_rref_python_annotation (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2075
2022-05-18T03:57:38.8489517Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2076
2022-05-18T03:57:38.8512379Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2077
2022-05-18T03:57:38.8536386Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2078
2022-05-18T03:57:39.5349470Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb1ubpwmn
2022-05-18T03:57:39.5350541Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb1ubpwmn/_remote_module_non_scriptable.py
2022-05-18T03:57:39.5709051Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv3vzch62
2022-05-18T03:57:39.5709828Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv3vzch62/_remote_module_non_scriptable.py
2022-05-18T03:57:39.5757343Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjxhn_kph
2022-05-18T03:57:39.5758830Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjxhn_kph/_remote_module_non_scriptable.py
2022-05-18T03:57:39.5775626Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdbp_kgpg
2022-05-18T03:57:39.5778083Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdbp_kgpg/_remote_module_non_scriptable.py
2022-05-18T03:57:39.7838070Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:57:39.8190906Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:57:39.8224087Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:57:39.8275404Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:57:40.3578869Z ok (1.830s)
2022-05-18T03:57:40.3579134Z 
2022-05-18T03:57:40.3579659Z ----------------------------------------------------------------------
2022-05-18T03:57:40.3579904Z Ran 1 test in 1.831s
2022-05-18T03:57:40.3580037Z 
2022-05-18T03:57:40.3580098Z OK
2022-05-18T03:57:40.3580189Z 
2022-05-18T03:57:40.3580282Z Generating XML reports...
2022-05-18T03:57:40.3613683Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220518035738.xml
2022-05-18T03:57:41.1303765Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpynj6924c
2022-05-18T03:57:41.1304566Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpynj6924c/_remote_module_non_scriptable.py
2022-05-18T03:57:41.3807321Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T03:57:41.3817271Z 
2022-05-18T03:57:41.3817778Z Running tests...
2022-05-18T03:57:41.3818385Z ----------------------------------------------------------------------
2022-05-18T03:57:41.6966577Z   test_some_kwargs_are_populated_by_defaults (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2294
2022-05-18T03:57:41.6988850Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2295
2022-05-18T03:57:41.7011439Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2296
2022-05-18T03:57:41.7035719Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2297
2022-05-18T03:57:42.2694777Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk61z0csl
2022-05-18T03:57:42.2695582Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk61z0csl/_remote_module_non_scriptable.py
2022-05-18T03:57:42.2758714Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpigj0haib
2022-05-18T03:57:42.2759919Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpigj0haib/_remote_module_non_scriptable.py
2022-05-18T03:57:42.3293616Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp82poxttz
2022-05-18T03:57:42.3294463Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp82poxttz/_remote_module_non_scriptable.py
2022-05-18T03:57:42.3345598Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx7gmyu7b
2022-05-18T03:57:42.3346835Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx7gmyu7b/_remote_module_non_scriptable.py
2022-05-18T03:57:42.5208520Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:57:42.5251730Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:57:42.5797488Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:57:42.5822812Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:57:43.2078540Z ok (1.826s)
2022-05-18T03:57:43.2078720Z 
2022-05-18T03:57:43.2079133Z ----------------------------------------------------------------------
2022-05-18T03:57:43.2079387Z Ran 1 test in 1.826s
2022-05-18T03:57:43.2079516Z 
2022-05-18T03:57:43.2079565Z OK
2022-05-18T03:57:43.2079659Z 
2022-05-18T03:57:43.2079751Z Generating XML reports...
2022-05-18T03:57:43.2114263Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220518035741.xml
2022-05-18T03:57:43.9875295Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcmq3u60p
2022-05-18T03:57:43.9876189Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcmq3u60p/_remote_module_non_scriptable.py
2022-05-18T03:57:44.2411345Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T03:57:44.2420951Z 
2022-05-18T03:57:44.2421085Z Running tests...
2022-05-18T03:57:44.2421720Z ----------------------------------------------------------------------
2022-05-18T03:57:44.5581507Z   test_torchscript_function (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2517
2022-05-18T03:57:44.5604956Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2518
2022-05-18T03:57:44.5628130Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2519
2022-05-18T03:57:44.5652133Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2520
2022-05-18T03:57:45.1846584Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6wmvvcfe
2022-05-18T03:57:45.1847364Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6wmvvcfe/_remote_module_non_scriptable.py
2022-05-18T03:57:45.1894149Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnq_0vp5p
2022-05-18T03:57:45.1896151Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnq_0vp5p/_remote_module_non_scriptable.py
2022-05-18T03:57:45.2223403Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsamvufj_
2022-05-18T03:57:45.2224752Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsamvufj_/_remote_module_non_scriptable.py
2022-05-18T03:57:45.2314321Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsdwn9p9b
2022-05-18T03:57:45.2315994Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsdwn9p9b/_remote_module_non_scriptable.py
2022-05-18T03:57:45.4360802Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:57:45.4383706Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:57:45.4714452Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:57:45.4790583Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:57:45.9694648Z ok (1.727s)
2022-05-18T03:57:45.9694942Z 
2022-05-18T03:57:45.9695455Z ----------------------------------------------------------------------
2022-05-18T03:57:45.9695876Z Ran 1 test in 1.727s
2022-05-18T03:57:45.9695995Z 
2022-05-18T03:57:45.9696059Z OK
2022-05-18T03:57:45.9696154Z 
2022-05-18T03:57:45.9696247Z Generating XML reports...
2022-05-18T03:57:45.9729580Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220518035744.xml
2022-05-18T03:57:46.7553749Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_z1wepie
2022-05-18T03:57:46.7554668Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_z1wepie/_remote_module_non_scriptable.py
2022-05-18T03:57:47.0092564Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T03:57:47.0102557Z 
2022-05-18T03:57:47.0110719Z Running tests...
2022-05-18T03:57:47.0111239Z ----------------------------------------------------------------------
2022-05-18T03:57:47.3279949Z   test_torchscript_function_exception (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2736
2022-05-18T03:57:47.3302383Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2737
2022-05-18T03:57:47.3325834Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2738
2022-05-18T03:57:47.3350056Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2739
2022-05-18T03:57:47.9423175Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdelxjya2
2022-05-18T03:57:47.9424450Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdelxjya2/_remote_module_non_scriptable.py
2022-05-18T03:57:47.9702524Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy5alff9_
2022-05-18T03:57:47.9703584Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy5alff9_/_remote_module_non_scriptable.py
2022-05-18T03:57:47.9729465Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzbb29k5c
2022-05-18T03:57:47.9731986Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzbb29k5c/_remote_module_non_scriptable.py
2022-05-18T03:57:47.9795858Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgwfnp2i7
2022-05-18T03:57:47.9797402Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgwfnp2i7/_remote_module_non_scriptable.py
2022-05-18T03:57:48.1917837Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:57:48.2165159Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:57:48.2228399Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:57:48.2269240Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:57:48.6389931Z ok (1.628s)
2022-05-18T03:57:48.6390190Z 
2022-05-18T03:57:48.6390684Z ----------------------------------------------------------------------
2022-05-18T03:57:48.6391115Z Ran 1 test in 1.629s
2022-05-18T03:57:48.6391481Z 
2022-05-18T03:57:48.6391530Z OK
2022-05-18T03:57:48.6391625Z 
2022-05-18T03:57:48.6391719Z Generating XML reports...
2022-05-18T03:57:48.6424905Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220518035747.xml
2022-05-18T03:57:49.4213649Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqtob3oj8
2022-05-18T03:57:49.4214615Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqtob3oj8/_remote_module_non_scriptable.py
2022-05-18T03:57:49.6771802Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T03:57:49.6781327Z 
2022-05-18T03:57:49.6781457Z Running tests...
2022-05-18T03:57:49.6782226Z ----------------------------------------------------------------------
2022-05-18T03:57:49.9966391Z   test_torchscript_functions_not_supported (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2955
2022-05-18T03:57:49.9989923Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2956
2022-05-18T03:57:50.0013736Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2957
2022-05-18T03:57:50.0038492Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2958
2022-05-18T03:57:50.6142645Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi05v9xab
2022-05-18T03:57:50.6143603Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi05v9xab/_remote_module_non_scriptable.py
2022-05-18T03:57:50.6537056Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9qwb6ed_
2022-05-18T03:57:50.6538165Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9qwb6ed_/_remote_module_non_scriptable.py
2022-05-18T03:57:50.6632191Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwrs1_3af
2022-05-18T03:57:50.6633406Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwrs1_3af/_remote_module_non_scriptable.py
2022-05-18T03:57:50.6688405Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpaybpsp68
2022-05-18T03:57:50.6689722Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpaybpsp68/_remote_module_non_scriptable.py
2022-05-18T03:57:50.8638708Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:57:50.9043624Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:57:50.9112046Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:57:50.9199356Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:57:51.1585381Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T03:57:51.1586255Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T03:57:51.1684298Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T03:57:51.1688930Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T03:57:51.1690567Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:57:51.1789207Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:57:51.1790426Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:57:51.1791595Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T03:57:51.5080292Z ok (1.830s)
2022-05-18T03:57:51.5080527Z 
2022-05-18T03:57:51.5081223Z ----------------------------------------------------------------------
2022-05-18T03:57:51.5081638Z Ran 1 test in 1.830s
2022-05-18T03:57:51.5081815Z 
2022-05-18T03:57:51.5081893Z OK
2022-05-18T03:57:51.5082040Z 
2022-05-18T03:57:51.5082178Z Generating XML reports...
2022-05-18T03:57:51.5116728Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220518035749.xml
2022-05-18T03:57:52.2872143Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7t403hik
2022-05-18T03:57:52.2872709Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7t403hik/_remote_module_non_scriptable.py
2022-05-18T03:57:52.5429342Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T03:57:52.5438884Z 
2022-05-18T03:57:52.5438975Z Running tests...
2022-05-18T03:57:52.5439526Z ----------------------------------------------------------------------
2022-05-18T03:57:52.8618705Z   test_unexepected_kwarg_is_specified (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3186
2022-05-18T03:57:52.8641769Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3187
2022-05-18T03:57:52.8665210Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3188
2022-05-18T03:57:52.8689295Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3189
2022-05-18T03:57:53.4496562Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv9nn3t07
2022-05-18T03:57:53.4497732Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv9nn3t07/_remote_module_non_scriptable.py
2022-05-18T03:57:53.4762383Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2yx8ricm
2022-05-18T03:57:53.4763161Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2yx8ricm/_remote_module_non_scriptable.py
2022-05-18T03:57:53.4925806Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpju_pkvvn
2022-05-18T03:57:53.4926724Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpju_pkvvn/_remote_module_non_scriptable.py
2022-05-18T03:57:53.5224194Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpspjt0sq9
2022-05-18T03:57:53.5224962Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpspjt0sq9/_remote_module_non_scriptable.py
2022-05-18T03:57:53.6997362Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:57:53.7269331Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:57:53.7411442Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:57:53.7715770Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:57:54.2730802Z ok (1.729s)
2022-05-18T03:57:54.2731026Z 
2022-05-18T03:57:54.2731440Z ----------------------------------------------------------------------
2022-05-18T03:57:54.2731700Z Ran 1 test in 1.729s
2022-05-18T03:57:54.2731820Z 
2022-05-18T03:57:54.2731869Z OK
2022-05-18T03:57:54.2731978Z 
2022-05-18T03:57:54.2732112Z Generating XML reports...
2022-05-18T03:57:54.2765503Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220518035752.xml
2022-05-18T03:57:55.0569902Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpaoe_ho07
2022-05-18T03:57:55.0570825Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpaoe_ho07/_remote_module_non_scriptable.py
2022-05-18T03:57:55.3118543Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T03:57:55.3127683Z 
2022-05-18T03:57:55.3128149Z Running tests...
2022-05-18T03:57:55.3128621Z ----------------------------------------------------------------------
2022-05-18T03:57:55.6250396Z   test_user_rrefs_confirmed (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3405
2022-05-18T03:57:55.6273295Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3406
2022-05-18T03:57:55.6295587Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3407
2022-05-18T03:57:55.6319983Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3408
2022-05-18T03:57:56.2591194Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp5nlg2_5
2022-05-18T03:57:56.2592017Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp5nlg2_5/_remote_module_non_scriptable.py
2022-05-18T03:57:56.2703676Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6y1qkvmz
2022-05-18T03:57:56.2705031Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6y1qkvmz/_remote_module_non_scriptable.py
2022-05-18T03:57:56.3285225Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7k__para
2022-05-18T03:57:56.3285983Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7k__para/_remote_module_non_scriptable.py
2022-05-18T03:57:56.3358714Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf7kbl5d7
2022-05-18T03:57:56.3360539Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf7kbl5d7/_remote_module_non_scriptable.py
2022-05-18T03:57:56.5093652Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:57:56.5186133Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:57:56.5770575Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:57:56.5820298Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:57:57.2364012Z ok (1.923s)
2022-05-18T03:57:57.2364170Z 
2022-05-18T03:57:57.2364504Z ----------------------------------------------------------------------
2022-05-18T03:57:57.2364760Z Ran 1 test in 1.924s
2022-05-18T03:57:57.2364877Z 
2022-05-18T03:57:57.2364926Z OK
2022-05-18T03:57:57.2365019Z 
2022-05-18T03:57:57.2365113Z Generating XML reports...
2022-05-18T03:57:57.2398686Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220518035755.xml
2022-05-18T03:57:58.0175357Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj1ee3q_g
2022-05-18T03:57:58.0176297Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj1ee3q_g/_remote_module_non_scriptable.py
2022-05-18T03:57:58.2688075Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T03:57:58.2698168Z 
2022-05-18T03:57:58.2698575Z Running tests...
2022-05-18T03:57:58.2698983Z ----------------------------------------------------------------------
2022-05-18T03:57:58.5833943Z   test_user_rrefs_confirmed_remote (__main__.TensorPipeJitRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3624
2022-05-18T03:57:58.5857600Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3625
2022-05-18T03:57:58.5881327Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3626
2022-05-18T03:57:58.5905206Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3627
2022-05-18T03:57:59.2024676Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2dade0po
2022-05-18T03:57:59.2025528Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2dade0po/_remote_module_non_scriptable.py
2022-05-18T03:57:59.2116458Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplcmr_d0f
2022-05-18T03:57:59.2117977Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplcmr_d0f/_remote_module_non_scriptable.py
2022-05-18T03:57:59.2210701Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjvri2xha
2022-05-18T03:57:59.2211935Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjvri2xha/_remote_module_non_scriptable.py
2022-05-18T03:57:59.2244327Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6173ekqb
2022-05-18T03:57:59.2246199Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6173ekqb/_remote_module_non_scriptable.py
2022-05-18T03:57:59.4498365Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:57:59.4590809Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:57:59.4676557Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:57:59.4685088Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:58:00.0949193Z ok (1.825s)
2022-05-18T03:58:00.0949441Z 
2022-05-18T03:58:00.0949992Z ----------------------------------------------------------------------
2022-05-18T03:58:00.0950299Z Ran 1 test in 1.825s
2022-05-18T03:58:00.0950419Z 
2022-05-18T03:58:00.0950481Z OK
2022-05-18T03:58:00.0950574Z 
2022-05-18T03:58:00.0950672Z Generating XML reports...
2022-05-18T03:58:00.0985480Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220518035758.xml
2022-05-18T03:58:00.8613144Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpadl0p68d
2022-05-18T03:58:00.8613690Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpadl0p68d/_remote_module_non_scriptable.py
2022-05-18T03:58:01.1124547Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T03:58:01.1133474Z 
2022-05-18T03:58:01.1133604Z Running tests...
2022-05-18T03:58:01.1134177Z ----------------------------------------------------------------------
2022-05-18T03:58:01.4344879Z   test_batch_updating_parameter_server (__main__.TensorPipeParameterServerTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3843
2022-05-18T03:58:01.4367023Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3844
2022-05-18T03:58:01.4389957Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3845
2022-05-18T03:58:01.4414142Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3846
2022-05-18T03:58:02.0165025Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr_le547d
2022-05-18T03:58:02.0165765Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr_le547d/_remote_module_non_scriptable.py
2022-05-18T03:58:02.0535123Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn3e14ih0
2022-05-18T03:58:02.0535950Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn3e14ih0/_remote_module_non_scriptable.py
2022-05-18T03:58:02.0703807Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkom6s5rs
2022-05-18T03:58:02.0704600Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkom6s5rs/_remote_module_non_scriptable.py
2022-05-18T03:58:02.0834713Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpoaqvpj7u
2022-05-18T03:58:02.0835661Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpoaqvpj7u/_remote_module_non_scriptable.py
2022-05-18T03:58:02.2629543Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:58:02.3017181Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:58:02.3198593Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:58:02.3335652Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:58:02.5365453Z 03:58:02 Start training
2022-05-18T03:58:02.5405562Z 03:58:02 worker1 processing one batch
2022-05-18T03:58:02.5411956Z 03:58:02 worker3 processing one batch
2022-05-18T03:58:02.5417013Z 03:58:02 worker2 processing one batch
2022-05-18T03:58:02.5417600Z 03:58:02 worker1 reporting grads
2022-05-18T03:58:02.5422005Z 03:58:02 worker2 reporting grads
2022-05-18T03:58:02.5423068Z 03:58:02 worker3 reporting grads
2022-05-18T03:58:02.5431824Z 03:58:02 PS got 0/3 updates
2022-05-18T03:58:02.5435888Z 03:58:02 PS got 1/3 updates
2022-05-18T03:58:02.5437612Z 03:58:02 PS got 2/3 updates
2022-05-18T03:58:02.5445885Z 03:58:02 PS updated model
2022-05-18T03:58:02.5449211Z 03:58:02 worker1 got updated model
2022-05-18T03:58:02.5450997Z 03:58:02 worker1 processing one batch
2022-05-18T03:58:02.5453748Z 03:58:02 worker3 got updated model
2022-05-18T03:58:02.5455732Z 03:58:02 worker3 processing one batch
2022-05-18T03:58:02.5456157Z 03:58:02 worker1 reporting grads
2022-05-18T03:58:02.5460018Z 03:58:02 worker3 reporting grads
2022-05-18T03:58:02.5461498Z 03:58:02 PS got 0/3 updates
2022-05-18T03:58:02.5468594Z 03:58:02 worker2 got updated model
2022-05-18T03:58:02.5469705Z 03:58:02 worker2 processing one batch
2022-05-18T03:58:02.5473142Z 03:58:02 worker2 reporting grads
2022-05-18T03:58:02.5477324Z 03:58:02 PS got 1/3 updates
2022-05-18T03:58:02.5480873Z 03:58:02 PS got 2/3 updates
2022-05-18T03:58:02.5492068Z 03:58:02 PS updated model
2022-05-18T03:58:02.5492573Z 03:58:02 worker1 got updated model
2022-05-18T03:58:02.5495605Z 03:58:02 worker1 processing one batch
2022-05-18T03:58:02.5499057Z 03:58:02 worker3 got updated model
2022-05-18T03:58:02.5502058Z 03:58:02 worker1 reporting grads
2022-05-18T03:58:02.5505143Z 03:58:02 worker3 processing one batch
2022-05-18T03:58:02.5506175Z 03:58:02 worker2 got updated model
2022-05-18T03:58:02.5506390Z 03:58:02 worker2 processing one batch
2022-05-18T03:58:02.5509559Z 03:58:02 worker3 reporting grads
2022-05-18T03:58:02.5509770Z 03:58:02 worker2 reporting grads
2022-05-18T03:58:02.5518942Z 03:58:02 PS got 0/3 updates
2022-05-18T03:58:02.5522392Z 03:58:02 PS got 1/3 updates
2022-05-18T03:58:02.5523444Z 03:58:02 PS got 2/3 updates
2022-05-18T03:58:02.5531666Z 03:58:02 PS updated model
2022-05-18T03:58:02.5535214Z 03:58:02 worker3 got updated model
2022-05-18T03:58:02.5536397Z 03:58:02 worker3 processing one batch
2022-05-18T03:58:02.5537041Z 03:58:02 worker3 reporting grads
2022-05-18T03:58:02.5537714Z 03:58:02 worker1 got updated model
2022-05-18T03:58:02.5538727Z 03:58:02 worker1 processing one batch
2022-05-18T03:58:02.5543215Z 03:58:02 worker1 reporting grads
2022-05-18T03:58:02.5546749Z 03:58:02 worker2 got updated model
2022-05-18T03:58:02.5547517Z 03:58:02 worker2 processing one batch
2022-05-18T03:58:02.5548731Z 03:58:02 PS got 0/3 updates
2022-05-18T03:58:02.5551200Z 03:58:02 worker2 reporting grads
2022-05-18T03:58:02.5560634Z 03:58:02 PS got 1/3 updates
2022-05-18T03:58:02.5561025Z 03:58:02 PS got 2/3 updates
2022-05-18T03:58:02.5572371Z 03:58:02 PS updated model
2022-05-18T03:58:02.5575031Z 03:58:02 worker3 got updated model
2022-05-18T03:58:02.5579442Z 03:58:02 worker1 got updated model
2022-05-18T03:58:02.5583495Z 03:58:02 worker2 got updated model
2022-05-18T03:58:02.5587017Z 03:58:02 Finish training
2022-05-18T03:58:02.5587422Z 03:58:02 Time spent training: 0.022403806000056647s
2022-05-18T03:58:02.7451213Z ok (1.631s)
2022-05-18T03:58:02.7451368Z 
2022-05-18T03:58:02.7452955Z ----------------------------------------------------------------------
2022-05-18T03:58:02.7453389Z Ran 1 test in 1.632s
2022-05-18T03:58:02.7453514Z 
2022-05-18T03:58:02.7453583Z OK
2022-05-18T03:58:02.7453678Z 
2022-05-18T03:58:02.7453778Z Generating XML reports...
2022-05-18T03:58:02.7488488Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeParameterServerTest-20220518035801.xml
2022-05-18T03:58:03.5056627Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb5vc5nin
2022-05-18T03:58:03.5057254Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb5vc5nin/_remote_module_non_scriptable.py
2022-05-18T03:58:03.7562702Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T03:58:03.7572592Z 
2022-05-18T03:58:03.7572860Z Running tests...
2022-05-18T03:58:03.7573252Z ----------------------------------------------------------------------
2022-05-18T03:58:04.0686391Z   test_rl_rpc (__main__.TensorPipeReinforcementLearningRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4062
2022-05-18T03:58:04.0708819Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4063
2022-05-18T03:58:04.0732194Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4064
2022-05-18T03:58:04.0757629Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4065
2022-05-18T03:58:04.6464573Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6jqfoy0r
2022-05-18T03:58:04.6465575Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6jqfoy0r/_remote_module_non_scriptable.py
2022-05-18T03:58:04.6645455Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw54orovp
2022-05-18T03:58:04.6646532Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw54orovp/_remote_module_non_scriptable.py
2022-05-18T03:58:04.6905061Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp81jpeyi4
2022-05-18T03:58:04.6905906Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp81jpeyi4/_remote_module_non_scriptable.py
2022-05-18T03:58:04.7083095Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjwrm0jls
2022-05-18T03:58:04.7084529Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjwrm0jls/_remote_module_non_scriptable.py
2022-05-18T03:58:04.8929408Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:58:04.9126347Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:58:04.9381405Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:58:04.9559009Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:58:05.4071233Z Solved! Running reward is now 489.8051609821906!
2022-05-18T03:58:05.6800783Z ok (1.923s)
2022-05-18T03:58:05.6800986Z 
2022-05-18T03:58:05.6801444Z ----------------------------------------------------------------------
2022-05-18T03:58:05.6801747Z Ran 1 test in 1.923s
2022-05-18T03:58:05.6801899Z 
2022-05-18T03:58:05.6801993Z OK
2022-05-18T03:58:05.6802179Z 
2022-05-18T03:58:05.6802323Z Generating XML reports...
2022-05-18T03:58:05.6836055Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeReinforcementLearningRpcTest-20220518035803.xml
2022-05-18T03:58:06.4460073Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd5lkil3n
2022-05-18T03:58:06.4460763Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd5lkil3n/_remote_module_non_scriptable.py
2022-05-18T03:58:06.6981430Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T03:58:06.6991035Z 
2022-05-18T03:58:06.6991149Z Running tests...
2022-05-18T03:58:06.6991722Z ----------------------------------------------------------------------
2022-05-18T03:58:07.0126388Z   test_bad_module (__main__.TensorPipeRemoteModuleTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4284
2022-05-18T03:58:07.0150106Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4285
2022-05-18T03:58:07.5701243Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr6jfhbgz
2022-05-18T03:58:07.5702374Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr6jfhbgz/_remote_module_non_scriptable.py
2022-05-18T03:58:07.6030884Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuuowuvap
2022-05-18T03:58:07.6031726Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuuowuvap/_remote_module_non_scriptable.py
2022-05-18T03:58:07.8163366Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:58:07.8471401Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:58:07.9497533Z On WorkerInfo(id=1, name=worker1):
2022-05-18T03:58:07.9498292Z ValueError("Expect `module_cls(*args, **kwargs)` returns an instance of <class nn.Module>, but it returns an instance of <class 'torch.testing._internal.distributed.nn.api.remote_module_test.BadModule'>.")
2022-05-18T03:58:07.9498932Z Traceback (most recent call last):
2022-05-18T03:58:07.9499571Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function
2022-05-18T03:58:07.9499924Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-05-18T03:58:07.9500405Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/nn/api/remote_module.py", line 86, in _create_module
2022-05-18T03:58:07.9500781Z     "Expect `module_cls(*args, **kwargs)` returns an instance of <class nn.Module>, "
2022-05-18T03:58:07.9501414Z ValueError: Expect `module_cls(*args, **kwargs)` returns an instance of <class nn.Module>, but it returns an instance of <class 'torch.testing._internal.distributed.nn.api.remote_module_test.BadModule'>.
2022-05-18T03:58:07.9501775Z 
2022-05-18T03:58:07.9507956Z On WorkerInfo(id=1, name=worker1):
2022-05-18T03:58:07.9510949Z ValueError('On WorkerInfo(id=1, name=worker1):\nValueError("Expect `module_cls(*args, **kwargs)` returns an instance of <class nn.Module>, but it returns an instance of <class \'torch.testing._internal.distributed.nn.api.remote_module_test.BadModule\'>.")\nTraceback (most recent call last):\n  File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function\n    result = python_udf.func(*python_udf.args, **python_udf.kwargs)\n  File "/opt/conda/lib/python3.7/site-packages/torch/distributed/nn/api/remote_module.py", line 86, in _create_module\n    "Expect `module_cls(*args, **kwargs)` returns an instance of <class nn.Module>, "\nValueError: Expect `module_cls(*args, **kwargs)` returns an instance of <class nn.Module>, but it returns an instance of <class \'torch.testing._internal.distributed.nn.api.remote_module_test.BadModule\'>.\n')
2022-05-18T03:58:07.9512454Z Traceback (most recent call last):
2022-05-18T03:58:07.9513142Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function
2022-05-18T03:58:07.9513738Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-05-18T03:58:07.9514273Z   File "/tmp/tmpd5lkil3n/_remote_module_non_scriptable.py", line 47, in _remote_forward
2022-05-18T03:58:07.9514761Z     module = module_rref.local_value()
2022-05-18T03:58:07.9515508Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 220, in _handle_exception
2022-05-18T03:58:07.9516277Z     raise result.exception_type(result.msg.encode("utf-8").decode("unicode_escape"))
2022-05-18T03:58:07.9516804Z ValueError: On WorkerInfo(id=1, name=worker1):
2022-05-18T03:58:07.9517554Z ValueError("Expect `module_cls(*args, **kwargs)` returns an instance of <class nn.Module>, but it returns an instance of <class 'torch.testing._internal.distributed.nn.api.remote_module_test.BadModule'>.")
2022-05-18T03:58:07.9517938Z Traceback (most recent call last):
2022-05-18T03:58:07.9518322Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function
2022-05-18T03:58:07.9518654Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-05-18T03:58:07.9519280Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/nn/api/remote_module.py", line 86, in _create_module
2022-05-18T03:58:07.9519680Z     "Expect `module_cls(*args, **kwargs)` returns an instance of <class nn.Module>, "
2022-05-18T03:58:07.9520275Z ValueError: Expect `module_cls(*args, **kwargs)` returns an instance of <class nn.Module>, but it returns an instance of <class 'torch.testing._internal.distributed.nn.api.remote_module_test.BadModule'>.
2022-05-18T03:58:07.9520582Z 
2022-05-18T03:58:07.9520586Z 
2022-05-18T03:58:07.9526404Z On WorkerInfo(id=1, name=worker1):
2022-05-18T03:58:07.9527834Z ValueError("Expect `module_cls(*args, **kwargs)` returns an instance of <class nn.Module>, but it returns an instance of <class 'torch.testing._internal.distributed.nn.api.remote_module_test.BadModule'>.")
2022-05-18T03:58:07.9528728Z Traceback (most recent call last):
2022-05-18T03:58:07.9529862Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function
2022-05-18T03:58:07.9530500Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-05-18T03:58:07.9531075Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/nn/api/remote_module.py", line 86, in _create_module
2022-05-18T03:58:07.9531499Z     "Expect `module_cls(*args, **kwargs)` returns an instance of <class nn.Module>, "
2022-05-18T03:58:07.9532170Z ValueError: Expect `module_cls(*args, **kwargs)` returns an instance of <class nn.Module>, but it returns an instance of <class 'torch.testing._internal.distributed.nn.api.remote_module_test.BadModule'>.
2022-05-18T03:58:07.9532477Z 
2022-05-18T03:58:07.9532626Z On WorkerInfo(id=1, name=worker1):
2022-05-18T03:58:07.9534395Z ValueError('On WorkerInfo(id=1, name=worker1):\nValueError("Expect `module_cls(*args, **kwargs)` returns an instance of <class nn.Module>, but it returns an instance of <class \'torch.testing._internal.distributed.nn.api.remote_module_test.BadModule\'>.")\nTraceback (most recent call last):\n  File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function\n    result = python_udf.func(*python_udf.args, **python_udf.kwargs)\n  File "/opt/conda/lib/python3.7/site-packages/torch/distributed/nn/api/remote_module.py", line 86, in _create_module\n    "Expect `module_cls(*args, **kwargs)` returns an instance of <class nn.Module>, "\nValueError: Expect `module_cls(*args, **kwargs)` returns an instance of <class nn.Module>, but it returns an instance of <class \'torch.testing._internal.distributed.nn.api.remote_module_test.BadModule\'>.\n')
2022-05-18T03:58:07.9535492Z Traceback (most recent call last):
2022-05-18T03:58:07.9535933Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function
2022-05-18T03:58:07.9536296Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-05-18T03:58:07.9536844Z   File "/tmp/tmpd5lkil3n/_remote_module_non_scriptable.py", line 47, in _remote_forward
2022-05-18T03:58:07.9537317Z     module = module_rref.local_value()
2022-05-18T03:58:07.9537718Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 220, in _handle_exception
2022-05-18T03:58:07.9538196Z     raise result.exception_type(result.msg.encode("utf-8").decode("unicode_escape"))
2022-05-18T03:58:07.9538495Z ValueError: On WorkerInfo(id=1, name=worker1):
2022-05-18T03:58:07.9539060Z ValueError("Expect `module_cls(*args, **kwargs)` returns an instance of <class nn.Module>, but it returns an instance of <class 'torch.testing._internal.distributed.nn.api.remote_module_test.BadModule'>.")
2022-05-18T03:58:07.9539453Z Traceback (most recent call last):
2022-05-18T03:58:07.9539873Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function
2022-05-18T03:58:07.9540389Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-05-18T03:58:07.9541054Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/nn/api/remote_module.py", line 86, in _create_module
2022-05-18T03:58:07.9541636Z     "Expect `module_cls(*args, **kwargs)` returns an instance of <class nn.Module>, "
2022-05-18T03:58:07.9542332Z ValueError: Expect `module_cls(*args, **kwargs)` returns an instance of <class nn.Module>, but it returns an instance of <class 'torch.testing._internal.distributed.nn.api.remote_module_test.BadModule'>.
2022-05-18T03:58:07.9542641Z 
2022-05-18T03:58:07.9542646Z 
2022-05-18T03:58:08.1178744Z ok (1.418s)
2022-05-18T03:58:08.1179001Z 
2022-05-18T03:58:08.1179438Z ----------------------------------------------------------------------
2022-05-18T03:58:08.1179842Z Ran 1 test in 1.419s
2022-05-18T03:58:08.1180006Z 
2022-05-18T03:58:08.1180098Z OK
2022-05-18T03:58:08.1180239Z 
2022-05-18T03:58:08.1180378Z Generating XML reports...
2022-05-18T03:58:08.1215236Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRemoteModuleTest-20220518035806.xml
2022-05-18T03:58:08.8680345Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpylq7qx8i
2022-05-18T03:58:08.8681106Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpylq7qx8i/_remote_module_non_scriptable.py
2022-05-18T03:58:09.1217650Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T03:58:09.1228060Z 
2022-05-18T03:58:09.1228456Z Running tests...
2022-05-18T03:58:09.1228868Z ----------------------------------------------------------------------
2022-05-18T03:58:09.4385036Z   test_forward_async (__main__.TensorPipeRemoteModuleTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4399
2022-05-18T03:58:09.4407163Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4400
2022-05-18T03:58:10.0010760Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbel8bpkb
2022-05-18T03:58:10.0011557Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbel8bpkb/_remote_module_non_scriptable.py
2022-05-18T03:58:10.0488187Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8g5hrdhn
2022-05-18T03:58:10.0489194Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8g5hrdhn/_remote_module_non_scriptable.py
2022-05-18T03:58:10.2466209Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:58:10.2954131Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:58:10.4158094Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8g5hrdhn/_remote_module___torch___torch_testing__internal_distributed_nn_api_remote_module_test_MyModuleInterface.py
2022-05-18T03:58:10.4159420Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbel8bpkb/_remote_module___torch___torch_testing__internal_distributed_nn_api_remote_module_test_MyModuleInterface.py
2022-05-18T03:58:10.4211682Z INFO:torch.distributed.nn.jit.instantiator:Skipped writing /tmp/tmp8g5hrdhn/_remote_module___torch___torch_testing__internal_distributed_nn_api_remote_module_test_MyModuleInterface.py
2022-05-18T03:58:10.7439779Z ok (1.621s)
2022-05-18T03:58:10.7440052Z 
2022-05-18T03:58:10.7440619Z ----------------------------------------------------------------------
2022-05-18T03:58:10.7440904Z Ran 1 test in 1.621s
2022-05-18T03:58:10.7441021Z 
2022-05-18T03:58:10.7441069Z OK
2022-05-18T03:58:10.7441162Z 
2022-05-18T03:58:10.7441258Z Generating XML reports...
2022-05-18T03:58:10.7475539Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRemoteModuleTest-20220518035809.xml
2022-05-18T03:58:11.5261993Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplclo9i1c
2022-05-18T03:58:11.5262727Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplclo9i1c/_remote_module_non_scriptable.py
2022-05-18T03:58:11.7774878Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T03:58:11.7785035Z 
2022-05-18T03:58:11.7785538Z Running tests...
2022-05-18T03:58:11.7785985Z ----------------------------------------------------------------------
2022-05-18T03:58:12.0928680Z   test_forward_async_script (__main__.TensorPipeRemoteModuleTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4514
2022-05-18T03:58:12.0951958Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4515
2022-05-18T03:58:12.6651440Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpngy5qchb
2022-05-18T03:58:12.6652189Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpngy5qchb/_remote_module_non_scriptable.py
2022-05-18T03:58:12.6946626Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2rs0b48_
2022-05-18T03:58:12.6947371Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2rs0b48_/_remote_module_non_scriptable.py
2022-05-18T03:58:12.9110941Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:58:12.9412606Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:58:13.0666046Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpngy5qchb/_remote_module___torch___torch_testing__internal_distributed_nn_api_remote_module_test_MyModuleInterface.py
2022-05-18T03:58:13.0667319Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2rs0b48_/_remote_module___torch___torch_testing__internal_distributed_nn_api_remote_module_test_MyModuleInterface.py
2022-05-18T03:58:13.0699692Z INFO:torch.distributed.nn.jit.instantiator:Skipped writing /tmp/tmp2rs0b48_/_remote_module___torch___torch_testing__internal_distributed_nn_api_remote_module_test_MyModuleInterface.py
2022-05-18T03:58:13.3984714Z ok (1.620s)
2022-05-18T03:58:13.3984940Z 
2022-05-18T03:58:13.3985426Z ----------------------------------------------------------------------
2022-05-18T03:58:13.3985896Z Ran 1 test in 1.620s
2022-05-18T03:58:13.3986056Z 
2022-05-18T03:58:13.3986119Z OK
2022-05-18T03:58:13.3986212Z 
2022-05-18T03:58:13.3986304Z Generating XML reports...
2022-05-18T03:58:13.4019571Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRemoteModuleTest-20220518035811.xml
2022-05-18T03:58:14.1721966Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphehqaa2j
2022-05-18T03:58:14.1722602Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphehqaa2j/_remote_module_non_scriptable.py
2022-05-18T03:58:14.4245176Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T03:58:14.4255203Z 
2022-05-18T03:58:14.4255282Z Running tests...
2022-05-18T03:58:14.4256309Z ----------------------------------------------------------------------
2022-05-18T03:58:14.7390830Z   test_forward_sync (__main__.TensorPipeRemoteModuleTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4633
2022-05-18T03:58:14.7412978Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4634
2022-05-18T03:58:15.2895654Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5me2_3dn
2022-05-18T03:58:15.2896382Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5me2_3dn/_remote_module_non_scriptable.py
2022-05-18T03:58:15.2982128Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx8d1otkr
2022-05-18T03:58:15.2984462Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx8d1otkr/_remote_module_non_scriptable.py
2022-05-18T03:58:15.5333916Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:58:15.5451466Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:58:15.6571858Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5me2_3dn/_remote_module___torch___torch_testing__internal_distributed_nn_api_remote_module_test_MyModuleInterface.py
2022-05-18T03:58:15.6573857Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx8d1otkr/_remote_module___torch___torch_testing__internal_distributed_nn_api_remote_module_test_MyModuleInterface.py
2022-05-18T03:58:15.6623326Z INFO:torch.distributed.nn.jit.instantiator:Skipped writing /tmp/tmpx8d1otkr/_remote_module___torch___torch_testing__internal_distributed_nn_api_remote_module_test_MyModuleInterface.py
2022-05-18T03:58:15.9444280Z ok (1.519s)
2022-05-18T03:58:15.9444530Z 
2022-05-18T03:58:15.9444988Z ----------------------------------------------------------------------
2022-05-18T03:58:15.9445371Z Ran 1 test in 1.519s
2022-05-18T03:58:15.9445530Z 
2022-05-18T03:58:15.9445623Z OK
2022-05-18T03:58:15.9445774Z 
2022-05-18T03:58:15.9445918Z Generating XML reports...
2022-05-18T03:58:15.9480389Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRemoteModuleTest-20220518035814.xml
2022-05-18T03:58:16.7178209Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdz9n31l1
2022-05-18T03:58:16.7178733Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdz9n31l1/_remote_module_non_scriptable.py
2022-05-18T03:58:16.9708206Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T03:58:16.9717855Z 
2022-05-18T03:58:16.9718018Z Running tests...
2022-05-18T03:58:16.9718426Z ----------------------------------------------------------------------
2022-05-18T03:58:17.2867902Z   test_forward_sync_script (__main__.TensorPipeRemoteModuleTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4752
2022-05-18T03:58:17.2891525Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4753
2022-05-18T03:58:17.8422058Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp80u7bbj7
2022-05-18T03:58:17.8422754Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph4ty291b
2022-05-18T03:58:17.8423567Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp80u7bbj7/_remote_module_non_scriptable.py
2022-05-18T03:58:17.8424301Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph4ty291b/_remote_module_non_scriptable.py
2022-05-18T03:58:18.0869185Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:58:18.0894057Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:58:18.2284633Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph4ty291b/_remote_module___torch___torch_testing__internal_distributed_nn_api_remote_module_test_MyModuleInterface.py
2022-05-18T03:58:18.2316325Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp80u7bbj7/_remote_module___torch___torch_testing__internal_distributed_nn_api_remote_module_test_MyModuleInterface.py
2022-05-18T03:58:18.2317590Z INFO:torch.distributed.nn.jit.instantiator:Skipped writing /tmp/tmp80u7bbj7/_remote_module___torch___torch_testing__internal_distributed_nn_api_remote_module_test_MyModuleInterface.py
2022-05-18T03:58:18.4922110Z ok (1.520s)
2022-05-18T03:58:18.4922352Z 
2022-05-18T03:58:18.4922823Z ----------------------------------------------------------------------
2022-05-18T03:58:18.4923206Z Ran 1 test in 1.520s
2022-05-18T03:58:18.4923391Z 
2022-05-18T03:58:18.4923485Z OK
2022-05-18T03:58:18.4923642Z 
2022-05-18T03:58:18.4923775Z Generating XML reports...
2022-05-18T03:58:18.4957555Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRemoteModuleTest-20220518035816.xml
2022-05-18T03:58:19.2575232Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5doko_km
2022-05-18T03:58:19.2575957Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5doko_km/_remote_module_non_scriptable.py
2022-05-18T03:58:19.5102798Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T03:58:19.5112460Z 
2022-05-18T03:58:19.5112595Z Running tests...
2022-05-18T03:58:19.5113340Z ----------------------------------------------------------------------
2022-05-18T03:58:19.8221706Z   test_forward_with_kwargs (__main__.TensorPipeRemoteModuleTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4871
2022-05-18T03:58:19.8244473Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4872
2022-05-18T03:58:20.3786935Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq9tkrkk3
2022-05-18T03:58:20.3787661Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq9tkrkk3/_remote_module_non_scriptable.py
2022-05-18T03:58:20.3791542Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_71497vl
2022-05-18T03:58:20.3793768Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_71497vl/_remote_module_non_scriptable.py
2022-05-18T03:58:20.6241934Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:58:20.6252987Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:58:20.9273860Z ok (1.416s)
2022-05-18T03:58:20.9274045Z 
2022-05-18T03:58:20.9274522Z ----------------------------------------------------------------------
2022-05-18T03:58:20.9274944Z Ran 1 test in 1.416s
2022-05-18T03:58:20.9275154Z 
2022-05-18T03:58:20.9275274Z OK
2022-05-18T03:58:20.9275432Z 
2022-05-18T03:58:20.9275573Z Generating XML reports...
2022-05-18T03:58:20.9308747Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRemoteModuleTest-20220518035819.xml
2022-05-18T03:58:21.6816791Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9io6jo8d
2022-05-18T03:58:21.6817863Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9io6jo8d/_remote_module_non_scriptable.py
2022-05-18T03:58:21.9347093Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T03:58:21.9356163Z 
2022-05-18T03:58:21.9356270Z Running tests...
2022-05-18T03:58:21.9356710Z ----------------------------------------------------------------------
2022-05-18T03:58:22.2557416Z   test_get_module_rref (__main__.TensorPipeRemoteModuleTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4986
2022-05-18T03:58:22.2580789Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4987
2022-05-18T03:58:22.8263615Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx2e6rigc
2022-05-18T03:58:22.8264405Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx2e6rigc/_remote_module_non_scriptable.py
2022-05-18T03:58:22.8275576Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfvhjcv7v
2022-05-18T03:58:22.8277283Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfvhjcv7v/_remote_module_non_scriptable.py
2022-05-18T03:58:23.0709954Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:58:23.0719112Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:58:23.3610920Z ok (1.425s)
2022-05-18T03:58:23.3611156Z 
2022-05-18T03:58:23.3611588Z ----------------------------------------------------------------------
2022-05-18T03:58:23.3612016Z Ran 1 test in 1.425s
2022-05-18T03:58:23.3612193Z 
2022-05-18T03:58:23.3612277Z OK
2022-05-18T03:58:23.3612423Z 
2022-05-18T03:58:23.3612562Z Generating XML reports...
2022-05-18T03:58:23.3646823Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRemoteModuleTest-20220518035821.xml
2022-05-18T03:58:24.1110961Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwtnqiqlr
2022-05-18T03:58:24.1111431Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwtnqiqlr/_remote_module_non_scriptable.py
2022-05-18T03:58:24.3632164Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T03:58:24.3641971Z 
2022-05-18T03:58:24.3642088Z Running tests...
2022-05-18T03:58:24.3642436Z ----------------------------------------------------------------------
2022-05-18T03:58:24.6770134Z   test_remote_module_py_pickle_not_supported (__main__.TensorPipeRemoteModuleTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5101
2022-05-18T03:58:24.6794201Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5102
2022-05-18T03:58:25.2354900Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplm35okts
2022-05-18T03:58:25.2355660Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplm35okts/_remote_module_non_scriptable.py
2022-05-18T03:58:25.2669398Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4qfysvbw
2022-05-18T03:58:25.2670378Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4qfysvbw/_remote_module_non_scriptable.py
2022-05-18T03:58:25.4806798Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:58:25.5138057Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:58:25.7823227Z ok (1.418s)
2022-05-18T03:58:25.7823482Z 
2022-05-18T03:58:25.7824005Z ----------------------------------------------------------------------
2022-05-18T03:58:25.7824320Z Ran 1 test in 1.418s
2022-05-18T03:58:25.7824436Z 
2022-05-18T03:58:25.7824484Z OK
2022-05-18T03:58:25.7824578Z 
2022-05-18T03:58:25.7824678Z Generating XML reports...
2022-05-18T03:58:25.7858428Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRemoteModuleTest-20220518035824.xml
2022-05-18T03:58:26.5333161Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzb2d5gx7
2022-05-18T03:58:26.5333736Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzb2d5gx7/_remote_module_non_scriptable.py
2022-05-18T03:58:26.7863689Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T03:58:26.7873315Z 
2022-05-18T03:58:26.7873460Z Running tests...
2022-05-18T03:58:26.7873873Z ----------------------------------------------------------------------
2022-05-18T03:58:27.1004777Z   test_remote_module_py_pickle_not_supported_script (__main__.TensorPipeRemoteModuleTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5216
2022-05-18T03:58:27.1027862Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5217
2022-05-18T03:58:27.6558510Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1b5ixeaz
2022-05-18T03:58:27.6559687Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1b5ixeaz/_remote_module_non_scriptable.py
2022-05-18T03:58:27.6953214Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl_d7ak20
2022-05-18T03:58:27.6954616Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl_d7ak20/_remote_module_non_scriptable.py
2022-05-18T03:58:27.9024206Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:58:27.9421068Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:58:28.0675351Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl_d7ak20/_remote_module___torch___torch_testing__internal_distributed_nn_api_remote_module_test_MyModuleInterface.py
2022-05-18T03:58:28.0723295Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1b5ixeaz/_remote_module___torch___torch_testing__internal_distributed_nn_api_remote_module_test_MyModuleInterface.py
2022-05-18T03:58:28.0724570Z INFO:torch.distributed.nn.jit.instantiator:Skipped writing /tmp/tmp1b5ixeaz/_remote_module___torch___torch_testing__internal_distributed_nn_api_remote_module_test_MyModuleInterface.py
2022-05-18T03:58:28.3059385Z ok (1.518s)
2022-05-18T03:58:28.3059575Z 
2022-05-18T03:58:28.3060395Z ----------------------------------------------------------------------
2022-05-18T03:58:28.3060651Z Ran 1 test in 1.519s
2022-05-18T03:58:28.3060768Z 
2022-05-18T03:58:28.3060831Z OK
2022-05-18T03:58:28.3060922Z 
2022-05-18T03:58:28.3061016Z Generating XML reports...
2022-05-18T03:58:28.3093906Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRemoteModuleTest-20220518035826.xml
2022-05-18T03:58:29.0824551Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph7k44ocs
2022-05-18T03:58:29.0825296Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph7k44ocs/_remote_module_non_scriptable.py
2022-05-18T03:58:29.3325612Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T03:58:29.3335063Z 
2022-05-18T03:58:29.3335153Z Running tests...
2022-05-18T03:58:29.3335603Z ----------------------------------------------------------------------
2022-05-18T03:58:29.6435003Z   test_remote_parameters (__main__.TensorPipeRemoteModuleTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5331
2022-05-18T03:58:29.6457929Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5332
2022-05-18T03:58:30.1937275Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphoogli_f
2022-05-18T03:58:30.1938041Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphoogli_f/_remote_module_non_scriptable.py
2022-05-18T03:58:30.2024304Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsk4hlyb1
2022-05-18T03:58:30.2026402Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsk4hlyb1/_remote_module_non_scriptable.py
2022-05-18T03:58:30.4374742Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:58:30.4482280Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:58:30.7486063Z ok (1.415s)
2022-05-18T03:58:30.7486340Z 
2022-05-18T03:58:30.7486680Z ----------------------------------------------------------------------
2022-05-18T03:58:30.7486952Z Ran 1 test in 1.415s
2022-05-18T03:58:30.7487067Z 
2022-05-18T03:58:30.7487129Z OK
2022-05-18T03:58:30.7487206Z 
2022-05-18T03:58:30.7487299Z Generating XML reports...
2022-05-18T03:58:30.7521062Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRemoteModuleTest-20220518035829.xml
2022-05-18T03:58:31.5041271Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj4ttm4q1
2022-05-18T03:58:31.5042481Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj4ttm4q1/_remote_module_non_scriptable.py
2022-05-18T03:58:31.7575205Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T03:58:31.7585191Z 
2022-05-18T03:58:31.7585653Z Running tests...
2022-05-18T03:58:31.7586057Z ----------------------------------------------------------------------
2022-05-18T03:58:32.0724761Z   test_send_remote_module_with_a_new_attribute_not_pickled_over_the_wire (__main__.TensorPipeRemoteModuleTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5446
2022-05-18T03:58:32.0748368Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5447
2022-05-18T03:58:32.6321770Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5moneaql
2022-05-18T03:58:32.6322540Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5moneaql/_remote_module_non_scriptable.py
2022-05-18T03:58:32.6339298Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplspwfmce
2022-05-18T03:58:32.6341139Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplspwfmce/_remote_module_non_scriptable.py
2022-05-18T03:58:32.8781887Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:58:32.8819363Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:58:32.9954565Z The new attribute ``new_attr`` of RemoteModule is ignored during RPC pickling. To pickle this attribute, please add it to ``_REMOTE_MODULE_PICKLED_ATTRIBUTES``. Otherwise, please explicitly add it to ``_REMOTE_MODULE_ATTRIBUTES_IGNORE_FOR_PICKLING``.
2022-05-18T03:58:33.1777988Z ok (1.419s)
2022-05-18T03:58:33.1778241Z 
2022-05-18T03:58:33.1778687Z ----------------------------------------------------------------------
2022-05-18T03:58:33.1779064Z Ran 1 test in 1.419s
2022-05-18T03:58:33.1779236Z 
2022-05-18T03:58:33.1779339Z OK
2022-05-18T03:58:33.1780120Z 
2022-05-18T03:58:33.1780274Z Generating XML reports...
2022-05-18T03:58:33.1814583Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRemoteModuleTest-20220518035831.xml
2022-05-18T03:58:33.9447645Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp780mm4fs
2022-05-18T03:58:33.9448169Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp780mm4fs/_remote_module_non_scriptable.py
2022-05-18T03:58:34.2002402Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T03:58:34.2012065Z 
2022-05-18T03:58:34.2012195Z Running tests...
2022-05-18T03:58:34.2012778Z ----------------------------------------------------------------------
2022-05-18T03:58:34.5227456Z   test_train_eval (__main__.TensorPipeRemoteModuleTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5561
2022-05-18T03:58:34.5249342Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5562
2022-05-18T03:58:35.0858304Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb8986yx3
2022-05-18T03:58:35.0858768Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb8986yx3/_remote_module_non_scriptable.py
2022-05-18T03:58:35.0864466Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6mh5u6_6
2022-05-18T03:58:35.0866579Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6mh5u6_6/_remote_module_non_scriptable.py
2022-05-18T03:58:35.3292885Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:58:35.3302786Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:58:35.6279049Z ok (1.426s)
2022-05-18T03:58:35.6279295Z 
2022-05-18T03:58:35.6279761Z ----------------------------------------------------------------------
2022-05-18T03:58:35.6280217Z Ran 1 test in 1.427s
2022-05-18T03:58:35.6280400Z 
2022-05-18T03:58:35.6280461Z OK
2022-05-18T03:58:35.6280552Z 
2022-05-18T03:58:35.6280835Z Generating XML reports...
2022-05-18T03:58:35.6314779Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRemoteModuleTest-20220518035834.xml
2022-05-18T03:58:36.3796158Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo6h1f509
2022-05-18T03:58:36.3797099Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo6h1f509/_remote_module_non_scriptable.py
2022-05-18T03:58:36.6325514Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T03:58:36.6335603Z 
2022-05-18T03:58:36.6335746Z Running tests...
2022-05-18T03:58:36.6336170Z ----------------------------------------------------------------------
2022-05-18T03:58:36.9497568Z   test_unsupported_methods (__main__.TensorPipeRemoteModuleTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5676
2022-05-18T03:58:36.9520054Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5677
2022-05-18T03:58:37.5244386Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8gc66xr8
2022-05-18T03:58:37.5245416Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8gc66xr8/_remote_module_non_scriptable.py
2022-05-18T03:58:37.5355599Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4iztlz0x
2022-05-18T03:58:37.5357615Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4iztlz0x/_remote_module_non_scriptable.py
2022-05-18T03:58:37.7692689Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:58:37.7800036Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:58:38.1551160Z ok (1.521s)
2022-05-18T03:58:38.1551401Z 
2022-05-18T03:58:38.1551854Z ----------------------------------------------------------------------
2022-05-18T03:58:38.1552226Z Ran 1 test in 1.521s
2022-05-18T03:58:38.1552417Z 
2022-05-18T03:58:38.1552510Z OK
2022-05-18T03:58:38.1552672Z 
2022-05-18T03:58:38.1552817Z Generating XML reports...
2022-05-18T03:58:38.1587786Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRemoteModuleTest-20220518035836.xml
2022-05-18T03:58:38.9305054Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxjvlta1r
2022-05-18T03:58:38.9305686Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxjvlta1r/_remote_module_non_scriptable.py
2022-05-18T03:58:39.1846418Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T03:58:39.1855908Z 
2022-05-18T03:58:39.1856000Z Running tests...
2022-05-18T03:58:39.1857292Z ----------------------------------------------------------------------
2022-05-18T03:58:39.5016589Z   test_add (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5791
2022-05-18T03:58:39.5040452Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5792
2022-05-18T03:58:39.5063411Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5793
2022-05-18T03:58:39.5087338Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5794
2022-05-18T03:58:40.0844808Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfns9gapa
2022-05-18T03:58:40.0846263Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfns9gapa/_remote_module_non_scriptable.py
2022-05-18T03:58:40.1079170Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqbedzu44
2022-05-18T03:58:40.1080295Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqbedzu44/_remote_module_non_scriptable.py
2022-05-18T03:58:40.1220117Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzq876qc3
2022-05-18T03:58:40.1221156Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzq876qc3/_remote_module_non_scriptable.py
2022-05-18T03:58:40.1520058Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj7by_c7y
2022-05-18T03:58:40.1520882Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj7by_c7y/_remote_module_non_scriptable.py
2022-05-18T03:58:40.3336458Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:58:40.3550172Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:58:40.3711422Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:58:40.3999644Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:58:40.9126943Z ok (1.727s)
2022-05-18T03:58:40.9127361Z 
2022-05-18T03:58:40.9127893Z ----------------------------------------------------------------------
2022-05-18T03:58:40.9128180Z Ran 1 test in 1.727s
2022-05-18T03:58:40.9128295Z 
2022-05-18T03:58:40.9128357Z OK
2022-05-18T03:58:40.9128450Z 
2022-05-18T03:58:40.9128764Z Generating XML reports...
2022-05-18T03:58:40.9162316Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518035839.xml
2022-05-18T03:58:41.6823616Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr922mf4i
2022-05-18T03:58:41.6824166Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr922mf4i/_remote_module_non_scriptable.py
2022-05-18T03:58:41.9351169Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T03:58:41.9359889Z 
2022-05-18T03:58:41.9359987Z Running tests...
2022-05-18T03:58:41.9360425Z ----------------------------------------------------------------------
2022-05-18T03:58:42.2521716Z   test_add_done_callback (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6010
2022-05-18T03:58:42.2545148Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6011
2022-05-18T03:58:42.2568539Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6012
2022-05-18T03:58:42.2592623Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6013
2022-05-18T03:58:42.8733776Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsfse893b
2022-05-18T03:58:42.8734725Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsfse893b/_remote_module_non_scriptable.py
2022-05-18T03:58:42.8802555Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7x5egyye
2022-05-18T03:58:42.8804001Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7x5egyye/_remote_module_non_scriptable.py
2022-05-18T03:58:42.8873505Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4xdkj47t
2022-05-18T03:58:42.8874845Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4xdkj47t/_remote_module_non_scriptable.py
2022-05-18T03:58:42.8945173Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc_wdj3qa
2022-05-18T03:58:42.8947360Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc_wdj3qa/_remote_module_non_scriptable.py
2022-05-18T03:58:43.1225552Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:58:43.1273648Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:58:43.1324092Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:58:43.1421700Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:58:43.6632670Z ok (1.727s)
2022-05-18T03:58:43.6632957Z 
2022-05-18T03:58:43.6633468Z ----------------------------------------------------------------------
2022-05-18T03:58:43.6633882Z Ran 1 test in 1.727s
2022-05-18T03:58:43.6633998Z 
2022-05-18T03:58:43.6634057Z OK
2022-05-18T03:58:43.6634136Z 
2022-05-18T03:58:43.6634247Z Generating XML reports...
2022-05-18T03:58:43.6669332Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518035841.xml
2022-05-18T03:58:44.4393949Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq5hbtglb
2022-05-18T03:58:44.4394604Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq5hbtglb/_remote_module_non_scriptable.py
2022-05-18T03:58:44.6913709Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T03:58:44.6923151Z 
2022-05-18T03:58:44.6923252Z Running tests...
2022-05-18T03:58:44.6923821Z ----------------------------------------------------------------------
2022-05-18T03:58:45.0068525Z   test_add_with_id (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6229
2022-05-18T03:58:45.0092407Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6230
2022-05-18T03:58:45.0115564Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6231
2022-05-18T03:58:45.0140201Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6232
2022-05-18T03:58:45.7065668Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_2lr9o2h
2022-05-18T03:58:45.7066471Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_2lr9o2h/_remote_module_non_scriptable.py
2022-05-18T03:58:45.7191751Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa5rsa5_k
2022-05-18T03:58:45.7193646Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa5rsa5_k/_remote_module_non_scriptable.py
2022-05-18T03:58:45.7207953Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdwe2w19f
2022-05-18T03:58:45.7210510Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdwe2w19f/_remote_module_non_scriptable.py
2022-05-18T03:58:45.7490241Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu_b75wni
2022-05-18T03:58:45.7491529Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu_b75wni/_remote_module_non_scriptable.py
2022-05-18T03:58:45.9524150Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:58:45.9660016Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:58:45.9681891Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:58:45.9969657Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:58:46.4181475Z ok (1.725s)
2022-05-18T03:58:46.4181703Z 
2022-05-18T03:58:46.4182013Z ----------------------------------------------------------------------
2022-05-18T03:58:46.4182268Z Ran 1 test in 1.726s
2022-05-18T03:58:46.4182370Z 
2022-05-18T03:58:46.4182435Z OK
2022-05-18T03:58:46.4182528Z 
2022-05-18T03:58:46.4182628Z Generating XML reports...
2022-05-18T03:58:46.4216145Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518035844.xml
2022-05-18T03:58:47.1830990Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1cgrp1yk
2022-05-18T03:58:47.1831670Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1cgrp1yk/_remote_module_non_scriptable.py
2022-05-18T03:58:47.4364484Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T03:58:47.4374130Z 
2022-05-18T03:58:47.4374254Z Running tests...
2022-05-18T03:58:47.4375227Z ----------------------------------------------------------------------
2022-05-18T03:58:47.7501085Z   test_all_gather (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6448
2022-05-18T03:58:47.7524196Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6449
2022-05-18T03:58:47.7547104Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6450
2022-05-18T03:58:47.7570741Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6451
2022-05-18T03:58:48.4036213Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkvqk_ui_
2022-05-18T03:58:48.4037005Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkvqk_ui_/_remote_module_non_scriptable.py
2022-05-18T03:58:48.4102735Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmp26m3b5
2022-05-18T03:58:48.4103755Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmp26m3b5/_remote_module_non_scriptable.py
2022-05-18T03:58:48.4290146Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8kvqegd6
2022-05-18T03:58:48.4291248Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8kvqegd6/_remote_module_non_scriptable.py
2022-05-18T03:58:48.4624612Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd0ft30tf
2022-05-18T03:58:48.4625711Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd0ft30tf/_remote_module_non_scriptable.py
2022-05-18T03:58:48.6521617Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:58:48.6576681Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:58:48.6752826Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:58:48.7103550Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:58:49.1610675Z ok (1.723s)
2022-05-18T03:58:49.1610973Z 
2022-05-18T03:58:49.1611516Z ----------------------------------------------------------------------
2022-05-18T03:58:49.1611873Z Ran 1 test in 1.724s
2022-05-18T03:58:49.1611988Z 
2022-05-18T03:58:49.1612050Z OK
2022-05-18T03:58:49.1612129Z 
2022-05-18T03:58:49.1612222Z Generating XML reports...
2022-05-18T03:58:49.1648964Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518035847.xml
2022-05-18T03:58:49.9358272Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxebsk95b
2022-05-18T03:58:49.9358740Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxebsk95b/_remote_module_non_scriptable.py
2022-05-18T03:58:50.1888431Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T03:58:50.1897907Z 
2022-05-18T03:58:50.1898041Z Running tests...
2022-05-18T03:58:50.1898620Z ----------------------------------------------------------------------
2022-05-18T03:58:50.5034637Z   test_all_gather_timeout (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6667
2022-05-18T03:58:50.5056098Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6668
2022-05-18T03:58:50.5079320Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6669
2022-05-18T03:58:50.5103666Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6670
2022-05-18T03:58:51.1941648Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr9kikbjx
2022-05-18T03:58:51.1942450Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr9kikbjx/_remote_module_non_scriptable.py
2022-05-18T03:58:51.2195219Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmbwqv_5p
2022-05-18T03:58:51.2195987Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmbwqv_5p/_remote_module_non_scriptable.py
2022-05-18T03:58:51.2506881Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmrxb3qyv
2022-05-18T03:58:51.2507916Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmrxb3qyv/_remote_module_non_scriptable.py
2022-05-18T03:58:51.2580391Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdk5sxdro
2022-05-18T03:58:51.2581412Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdk5sxdro/_remote_module_non_scriptable.py
2022-05-18T03:58:51.4411269Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:58:51.4659361Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:58:51.4969077Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:58:51.5053027Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:58:58.8635216Z [W tensorpipe_agent.cpp:942] RPC agent for worker0 encountered error when reading incoming response from worker1: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2022-05-18T03:59:00.7441352Z [W tensorpipe_agent.cpp:627] RPC agent for worker1 won't send response to request #5 to worker0, as the agent is shutting down
2022-05-18T03:59:00.9280473Z ok (10.738s)
2022-05-18T03:59:00.9280719Z 
2022-05-18T03:59:00.9281147Z ----------------------------------------------------------------------
2022-05-18T03:59:00.9281538Z Ran 1 test in 10.738s
2022-05-18T03:59:00.9281990Z 
2022-05-18T03:59:00.9282071Z OK
2022-05-18T03:59:00.9282216Z 
2022-05-18T03:59:00.9282355Z Generating XML reports...
2022-05-18T03:59:00.9316995Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518035850.xml
2022-05-18T03:59:01.7023799Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_ylbrzs3
2022-05-18T03:59:01.7024848Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_ylbrzs3/_remote_module_non_scriptable.py
2022-05-18T03:59:01.9538131Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T03:59:01.9548003Z 
2022-05-18T03:59:01.9548090Z Running tests...
2022-05-18T03:59:01.9549227Z ----------------------------------------------------------------------
2022-05-18T03:59:02.2692255Z   test_async_add (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6886
2022-05-18T03:59:02.2715311Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6887
2022-05-18T03:59:02.2738941Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6888
2022-05-18T03:59:02.2763227Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6889
2022-05-18T03:59:02.9128894Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6d71p_3g
2022-05-18T03:59:02.9129631Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6d71p_3g/_remote_module_non_scriptable.py
2022-05-18T03:59:02.9247654Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdwozc_9_
2022-05-18T03:59:02.9248779Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdwozc_9_/_remote_module_non_scriptable.py
2022-05-18T03:59:02.9464888Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2ck0eb37
2022-05-18T03:59:02.9465845Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2ck0eb37/_remote_module_non_scriptable.py
2022-05-18T03:59:02.9889069Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvgpoweux
2022-05-18T03:59:02.9890052Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvgpoweux/_remote_module_non_scriptable.py
2022-05-18T03:59:03.1608360Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:59:03.1744996Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:59:03.1934399Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:59:03.2347229Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:59:03.6864737Z ok (1.731s)
2022-05-18T03:59:03.6865012Z 
2022-05-18T03:59:03.6865517Z ----------------------------------------------------------------------
2022-05-18T03:59:03.6865772Z Ran 1 test in 1.732s
2022-05-18T03:59:03.6865906Z 
2022-05-18T03:59:03.6865956Z OK
2022-05-18T03:59:03.6866047Z 
2022-05-18T03:59:03.6866141Z Generating XML reports...
2022-05-18T03:59:03.6899872Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518035901.xml
2022-05-18T03:59:04.4560215Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpol8bv7hx
2022-05-18T03:59:04.4560681Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpol8bv7hx/_remote_module_non_scriptable.py
2022-05-18T03:59:04.7091110Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T03:59:04.7101346Z 
2022-05-18T03:59:04.7101730Z Running tests...
2022-05-18T03:59:04.7102123Z ----------------------------------------------------------------------
2022-05-18T03:59:05.0220437Z   test_async_class_method (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7106
2022-05-18T03:59:05.0243520Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7107
2022-05-18T03:59:05.0266140Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7108
2022-05-18T03:59:05.0289808Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7109
2022-05-18T03:59:05.6200493Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcdw5mgjb
2022-05-18T03:59:05.6201268Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcdw5mgjb/_remote_module_non_scriptable.py
2022-05-18T03:59:05.6468343Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3p2v8otm
2022-05-18T03:59:05.6469433Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3p2v8otm/_remote_module_non_scriptable.py
2022-05-18T03:59:05.6717116Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfbuuylnp
2022-05-18T03:59:05.6718242Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfbuuylnp/_remote_module_non_scriptable.py
2022-05-18T03:59:05.6766569Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcy3zr_az
2022-05-18T03:59:05.6768603Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcy3zr_az/_remote_module_non_scriptable.py
2022-05-18T03:59:05.8797226Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:59:05.8951202Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:59:05.9191910Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:59:05.9361501Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:59:06.3329074Z ok (1.622s)
2022-05-18T03:59:06.3329308Z 
2022-05-18T03:59:06.3329718Z ----------------------------------------------------------------------
2022-05-18T03:59:06.3329984Z Ran 1 test in 1.623s
2022-05-18T03:59:06.3330120Z 
2022-05-18T03:59:06.3330182Z OK
2022-05-18T03:59:06.3330275Z 
2022-05-18T03:59:06.3330354Z Generating XML reports...
2022-05-18T03:59:06.3362866Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518035904.xml
2022-05-18T03:59:07.1043690Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprw54_chs
2022-05-18T03:59:07.1044193Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprw54_chs/_remote_module_non_scriptable.py
2022-05-18T03:59:07.3593342Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T03:59:07.3603389Z 
2022-05-18T03:59:07.3603660Z Running tests...
2022-05-18T03:59:07.3604304Z ----------------------------------------------------------------------
2022-05-18T03:59:07.6773023Z   test_async_class_method_remote (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7325
2022-05-18T03:59:07.6795912Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7326
2022-05-18T03:59:07.6819100Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7327
2022-05-18T03:59:07.6843111Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7328
2022-05-18T03:59:08.3484916Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2jrl1dvs
2022-05-18T03:59:08.3485706Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2jrl1dvs/_remote_module_non_scriptable.py
2022-05-18T03:59:08.3752987Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp08p391l0
2022-05-18T03:59:08.3753780Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp08p391l0/_remote_module_non_scriptable.py
2022-05-18T03:59:08.3834897Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphyqtj0vv
2022-05-18T03:59:08.3836245Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphyqtj0vv/_remote_module_non_scriptable.py
2022-05-18T03:59:08.3898554Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg5g6tlyz
2022-05-18T03:59:08.3899827Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg5g6tlyz/_remote_module_non_scriptable.py
2022-05-18T03:59:08.6012169Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:59:08.6279163Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:59:08.6325544Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:59:08.6438725Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:59:09.0883541Z ok (1.728s)
2022-05-18T03:59:09.0883756Z 
2022-05-18T03:59:09.0884080Z ----------------------------------------------------------------------
2022-05-18T03:59:09.0884353Z Ran 1 test in 1.728s
2022-05-18T03:59:09.0884468Z 
2022-05-18T03:59:09.0884545Z OK
2022-05-18T03:59:09.0884638Z 
2022-05-18T03:59:09.0884731Z Generating XML reports...
2022-05-18T03:59:09.0918980Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518035907.xml
2022-05-18T03:59:09.8545387Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmhjxyh4d
2022-05-18T03:59:09.8546409Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmhjxyh4d/_remote_module_non_scriptable.py
2022-05-18T03:59:10.1069490Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T03:59:10.1079134Z 
2022-05-18T03:59:10.1079370Z Running tests...
2022-05-18T03:59:10.1079806Z ----------------------------------------------------------------------
2022-05-18T03:59:10.4215426Z   test_async_class_rref_proxy (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7544
2022-05-18T03:59:10.4238177Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7545
2022-05-18T03:59:10.4261692Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7546
2022-05-18T03:59:10.4287090Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7547
2022-05-18T03:59:11.0314333Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzf560fy6
2022-05-18T03:59:11.0315512Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzf560fy6/_remote_module_non_scriptable.py
2022-05-18T03:59:11.0455700Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6_9qu6nl
2022-05-18T03:59:11.0457686Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6_9qu6nl/_remote_module_non_scriptable.py
2022-05-18T03:59:11.0570576Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv060k13r
2022-05-18T03:59:11.0571954Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv060k13r/_remote_module_non_scriptable.py
2022-05-18T03:59:11.0791774Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphkb28xmq
2022-05-18T03:59:11.0792720Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphkb28xmq/_remote_module_non_scriptable.py
2022-05-18T03:59:11.2814715Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:59:11.3039903Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:59:11.3054270Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:59:11.3256628Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:59:11.8325644Z ok (1.724s)
2022-05-18T03:59:11.8326558Z 
2022-05-18T03:59:11.8326963Z ----------------------------------------------------------------------
2022-05-18T03:59:11.8327297Z Ran 1 test in 1.725s
2022-05-18T03:59:11.8327459Z 
2022-05-18T03:59:11.8327626Z OK
2022-05-18T03:59:11.8327721Z 
2022-05-18T03:59:11.8327814Z Generating XML reports...
2022-05-18T03:59:11.8361669Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518035910.xml
2022-05-18T03:59:12.6024867Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqo7afbze
2022-05-18T03:59:12.6025639Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqo7afbze/_remote_module_non_scriptable.py
2022-05-18T03:59:12.8570667Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T03:59:12.8581329Z 
2022-05-18T03:59:12.8581601Z Running tests...
2022-05-18T03:59:12.8582255Z ----------------------------------------------------------------------
2022-05-18T03:59:13.1752576Z   test_async_class_rref_proxy_async (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7763
2022-05-18T03:59:13.1775781Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7764
2022-05-18T03:59:13.1799368Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7765
2022-05-18T03:59:13.1823796Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7766
2022-05-18T03:59:13.8754295Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjo0035x0
2022-05-18T03:59:13.8755052Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjo0035x0/_remote_module_non_scriptable.py
2022-05-18T03:59:13.8800821Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjbf5_krh
2022-05-18T03:59:13.8802427Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjbf5_krh/_remote_module_non_scriptable.py
2022-05-18T03:59:13.8945219Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyva4em9g
2022-05-18T03:59:13.8946849Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyva4em9g/_remote_module_non_scriptable.py
2022-05-18T03:59:13.9300893Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz9u2wf42
2022-05-18T03:59:13.9302168Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz9u2wf42/_remote_module_non_scriptable.py
2022-05-18T03:59:14.1258431Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:59:14.1269791Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:59:14.1415888Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:59:14.1797754Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:59:14.6866161Z ok (1.828s)
2022-05-18T03:59:14.6866435Z 
2022-05-18T03:59:14.6866955Z ----------------------------------------------------------------------
2022-05-18T03:59:14.6867384Z Ran 1 test in 1.828s
2022-05-18T03:59:14.6867501Z 
2022-05-18T03:59:14.6867574Z OK
2022-05-18T03:59:14.6867665Z 
2022-05-18T03:59:14.6867747Z Generating XML reports...
2022-05-18T03:59:14.6900887Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518035912.xml
2022-05-18T03:59:15.4568624Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc9xmlm4i
2022-05-18T03:59:15.4569373Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc9xmlm4i/_remote_module_non_scriptable.py
2022-05-18T03:59:15.7134101Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T03:59:15.7144595Z 
2022-05-18T03:59:15.7144933Z Running tests...
2022-05-18T03:59:15.7145311Z ----------------------------------------------------------------------
2022-05-18T03:59:16.0284357Z   test_async_class_rref_proxy_remote (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7982
2022-05-18T03:59:16.0307726Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7983
2022-05-18T03:59:16.0330939Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7984
2022-05-18T03:59:16.0355042Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7985
2022-05-18T03:59:16.6950259Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyam2syja
2022-05-18T03:59:16.6951035Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyam2syja/_remote_module_non_scriptable.py
2022-05-18T03:59:16.6987674Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt7a47kvt
2022-05-18T03:59:16.6989122Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt7a47kvt/_remote_module_non_scriptable.py
2022-05-18T03:59:16.7208659Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuvig1k71
2022-05-18T03:59:16.7209532Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuvig1k71/_remote_module_non_scriptable.py
2022-05-18T03:59:16.7440010Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmvs6nfeg
2022-05-18T03:59:16.7441045Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmvs6nfeg/_remote_module_non_scriptable.py
2022-05-18T03:59:16.9429642Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:59:16.9467255Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:59:16.9691378Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:59:16.9899243Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:59:17.4395627Z ok (1.725s)
2022-05-18T03:59:17.4395885Z 
2022-05-18T03:59:17.4396417Z ----------------------------------------------------------------------
2022-05-18T03:59:17.4396743Z Ran 1 test in 1.725s
2022-05-18T03:59:17.4396860Z 
2022-05-18T03:59:17.4396932Z OK
2022-05-18T03:59:17.4397013Z 
2022-05-18T03:59:17.4397105Z Generating XML reports...
2022-05-18T03:59:17.4430927Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518035915.xml
2022-05-18T03:59:18.2172794Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5wph_tce
2022-05-18T03:59:18.2174219Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5wph_tce/_remote_module_non_scriptable.py
2022-05-18T03:59:18.4686644Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T03:59:18.4697076Z 
2022-05-18T03:59:18.4697202Z Running tests...
2022-05-18T03:59:18.4697776Z ----------------------------------------------------------------------
2022-05-18T03:59:18.7809180Z   test_async_function_chained (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8201
2022-05-18T03:59:18.7832444Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8202
2022-05-18T03:59:18.7855082Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8203
2022-05-18T03:59:18.7880040Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8204
2022-05-18T03:59:19.4260533Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo2ixue1a
2022-05-18T03:59:19.4261291Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo2ixue1a/_remote_module_non_scriptable.py
2022-05-18T03:59:19.4351986Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyx88jdxk
2022-05-18T03:59:19.4353117Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyx88jdxk/_remote_module_non_scriptable.py
2022-05-18T03:59:19.4375063Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd3qml0ho
2022-05-18T03:59:19.4377101Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd3qml0ho/_remote_module_non_scriptable.py
2022-05-18T03:59:19.4466876Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp19o33g9c
2022-05-18T03:59:19.4468482Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp19o33g9c/_remote_module_non_scriptable.py
2022-05-18T03:59:19.6734756Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:59:19.6830966Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:59:19.6839395Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:59:19.6959109Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:59:20.0916930Z ok (1.622s)
2022-05-18T03:59:20.0917102Z 
2022-05-18T03:59:20.0917590Z ----------------------------------------------------------------------
2022-05-18T03:59:20.0918024Z Ran 1 test in 1.622s
2022-05-18T03:59:20.0918175Z 
2022-05-18T03:59:20.0918281Z OK
2022-05-18T03:59:20.0918381Z 
2022-05-18T03:59:20.0918475Z Generating XML reports...
2022-05-18T03:59:20.0951694Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518035918.xml
2022-05-18T03:59:20.8606419Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprhq_lzkz
2022-05-18T03:59:20.8607185Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprhq_lzkz/_remote_module_non_scriptable.py
2022-05-18T03:59:21.1128492Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T03:59:21.1138325Z 
2022-05-18T03:59:21.1138440Z Running tests...
2022-05-18T03:59:21.1139022Z ----------------------------------------------------------------------
2022-05-18T03:59:21.4295211Z   test_async_function_chained_remote (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8420
2022-05-18T03:59:21.4317605Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8421
2022-05-18T03:59:21.4341055Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8422
2022-05-18T03:59:21.4365215Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8423
2022-05-18T03:59:22.1388019Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp565ht9r1
2022-05-18T03:59:22.1388786Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp565ht9r1/_remote_module_non_scriptable.py
2022-05-18T03:59:22.1472742Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj12ff9w3
2022-05-18T03:59:22.1473971Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj12ff9w3/_remote_module_non_scriptable.py
2022-05-18T03:59:22.1618908Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg4oc5apt
2022-05-18T03:59:22.1620231Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg4oc5apt/_remote_module_non_scriptable.py
2022-05-18T03:59:22.1640317Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnz4iup5y
2022-05-18T03:59:22.1642243Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnz4iup5y/_remote_module_non_scriptable.py
2022-05-18T03:59:22.3886877Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:59:22.3965091Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:59:22.4094564Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:59:22.4142527Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:59:22.8404619Z ok (1.726s)
2022-05-18T03:59:22.8405102Z 
2022-05-18T03:59:22.8405444Z ----------------------------------------------------------------------
2022-05-18T03:59:22.8405701Z Ran 1 test in 1.727s
2022-05-18T03:59:22.8405853Z 
2022-05-18T03:59:22.8406017Z OK
2022-05-18T03:59:22.8406112Z 
2022-05-18T03:59:22.8406208Z Generating XML reports...
2022-05-18T03:59:22.8440169Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518035921.xml
2022-05-18T03:59:23.6099831Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3wfgnyn8
2022-05-18T03:59:23.6100727Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3wfgnyn8/_remote_module_non_scriptable.py
2022-05-18T03:59:23.8642467Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T03:59:23.8652161Z 
2022-05-18T03:59:23.8652380Z Running tests...
2022-05-18T03:59:23.8652757Z ----------------------------------------------------------------------
2022-05-18T03:59:24.1791551Z   test_async_function_multi_chained (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8639
2022-05-18T03:59:24.1814341Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8640
2022-05-18T03:59:24.1838029Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8641
2022-05-18T03:59:24.1862035Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8642
2022-05-18T03:59:24.7954331Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5as0mkcd
2022-05-18T03:59:24.7955114Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5as0mkcd/_remote_module_non_scriptable.py
2022-05-18T03:59:24.8014157Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5o1k7nh0
2022-05-18T03:59:24.8016575Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5o1k7nh0/_remote_module_non_scriptable.py
2022-05-18T03:59:24.8056282Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9rszn68t
2022-05-18T03:59:24.8058238Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9rszn68t/_remote_module_non_scriptable.py
2022-05-18T03:59:24.8060963Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk1smp4ml
2022-05-18T03:59:24.8063432Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk1smp4ml/_remote_module_non_scriptable.py
2022-05-18T03:59:25.0416033Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:59:25.0476676Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:59:25.0508512Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:59:25.0532600Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:59:25.5902538Z ok (1.725s)
2022-05-18T03:59:25.5902850Z 
2022-05-18T03:59:25.5903463Z ----------------------------------------------------------------------
2022-05-18T03:59:25.5903705Z Ran 1 test in 1.725s
2022-05-18T03:59:25.5903822Z 
2022-05-18T03:59:25.5903892Z OK
2022-05-18T03:59:25.5903983Z 
2022-05-18T03:59:25.5904076Z Generating XML reports...
2022-05-18T03:59:25.5936634Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518035923.xml
2022-05-18T03:59:26.3592571Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2tvb0dbo
2022-05-18T03:59:26.3593399Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2tvb0dbo/_remote_module_non_scriptable.py
2022-05-18T03:59:26.6107415Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T03:59:26.6116984Z 
2022-05-18T03:59:26.6117240Z Running tests...
2022-05-18T03:59:26.6117572Z ----------------------------------------------------------------------
2022-05-18T03:59:26.9233946Z   test_async_function_multi_chained_async (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8858
2022-05-18T03:59:26.9256294Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8859
2022-05-18T03:59:26.9279527Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8860
2022-05-18T03:59:26.9303574Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8861
2022-05-18T03:59:27.5847316Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgt88ajk_
2022-05-18T03:59:27.5848551Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgt88ajk_/_remote_module_non_scriptable.py
2022-05-18T03:59:27.6115729Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkpa8wfwp
2022-05-18T03:59:27.6116831Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkpa8wfwp/_remote_module_non_scriptable.py
2022-05-18T03:59:27.6337043Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2646xqt6
2022-05-18T03:59:27.6337841Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2646xqt6/_remote_module_non_scriptable.py
2022-05-18T03:59:27.6572747Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqe2bu72d
2022-05-18T03:59:27.6573570Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqe2bu72d/_remote_module_non_scriptable.py
2022-05-18T03:59:27.8337761Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:59:27.8571577Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:59:27.8810133Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:59:27.9072854Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:59:28.3343678Z ok (1.722s)
2022-05-18T03:59:28.3343917Z 
2022-05-18T03:59:28.3344353Z ----------------------------------------------------------------------
2022-05-18T03:59:28.3344771Z Ran 1 test in 1.723s
2022-05-18T03:59:28.3344947Z 
2022-05-18T03:59:28.3345046Z OK
2022-05-18T03:59:28.3345189Z 
2022-05-18T03:59:28.3345334Z Generating XML reports...
2022-05-18T03:59:28.3379274Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518035926.xml
2022-05-18T03:59:29.0994623Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6fj03nn7
2022-05-18T03:59:29.0995133Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6fj03nn7/_remote_module_non_scriptable.py
2022-05-18T03:59:29.3528849Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T03:59:29.3538746Z 
2022-05-18T03:59:29.3538858Z Running tests...
2022-05-18T03:59:29.3539416Z ----------------------------------------------------------------------
2022-05-18T03:59:29.6654786Z   test_async_function_multi_chained_remote (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9077
2022-05-18T03:59:29.6678986Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9078
2022-05-18T03:59:29.6702419Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9079
2022-05-18T03:59:29.6728508Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9080
2022-05-18T03:59:30.2750328Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqxgypcao
2022-05-18T03:59:30.2751080Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqxgypcao/_remote_module_non_scriptable.py
2022-05-18T03:59:30.2795228Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpoi6kjq4t
2022-05-18T03:59:30.2796502Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpoi6kjq4t/_remote_module_non_scriptable.py
2022-05-18T03:59:30.2850062Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxdlq08pe
2022-05-18T03:59:30.2851833Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxdlq08pe/_remote_module_non_scriptable.py
2022-05-18T03:59:30.3065953Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq4ih21us
2022-05-18T03:59:30.3067361Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq4ih21us/_remote_module_non_scriptable.py
2022-05-18T03:59:30.5232974Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:59:30.5268001Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:59:30.5303870Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:59:30.5534623Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:59:30.9765931Z ok (1.622s)
2022-05-18T03:59:30.9766145Z 
2022-05-18T03:59:30.9766657Z ----------------------------------------------------------------------
2022-05-18T03:59:30.9767107Z Ran 1 test in 1.623s
2022-05-18T03:59:30.9767236Z 
2022-05-18T03:59:30.9767303Z OK
2022-05-18T03:59:30.9767395Z 
2022-05-18T03:59:30.9767488Z Generating XML reports...
2022-05-18T03:59:30.9803440Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518035929.xml
2022-05-18T03:59:31.7543907Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp558l54ez
2022-05-18T03:59:31.7544726Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp558l54ez/_remote_module_non_scriptable.py
2022-05-18T03:59:32.0082019Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T03:59:32.0091682Z 
2022-05-18T03:59:32.0091828Z Running tests...
2022-05-18T03:59:32.0092437Z ----------------------------------------------------------------------
2022-05-18T03:59:32.3309877Z   test_async_function_multi_fanout (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9296
2022-05-18T03:59:32.3332397Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9297
2022-05-18T03:59:32.3355865Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9298
2022-05-18T03:59:32.3380159Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9299
2022-05-18T03:59:32.9164591Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpln414k73
2022-05-18T03:59:32.9165335Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpln414k73/_remote_module_non_scriptable.py
2022-05-18T03:59:32.9638288Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2bgl4gw8
2022-05-18T03:59:32.9639101Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2bgl4gw8/_remote_module_non_scriptable.py
2022-05-18T03:59:32.9672141Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz4m2lq0y
2022-05-18T03:59:32.9675217Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz4m2lq0y/_remote_module_non_scriptable.py
2022-05-18T03:59:32.9887505Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa5by4430
2022-05-18T03:59:32.9888865Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa5by4430/_remote_module_non_scriptable.py
2022-05-18T03:59:33.1634998Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:59:33.2113634Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:59:33.2156272Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:59:33.2360480Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:59:33.7421408Z ok (1.733s)
2022-05-18T03:59:33.7421650Z 
2022-05-18T03:59:33.7422193Z ----------------------------------------------------------------------
2022-05-18T03:59:33.7422506Z Ran 1 test in 1.733s
2022-05-18T03:59:33.7422709Z 
2022-05-18T03:59:33.7423134Z OK
2022-05-18T03:59:33.7423238Z 
2022-05-18T03:59:33.7423319Z Generating XML reports...
2022-05-18T03:59:33.7456874Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518035932.xml
2022-05-18T03:59:34.5120846Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnymqvbmx
2022-05-18T03:59:34.5121464Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnymqvbmx/_remote_module_non_scriptable.py
2022-05-18T03:59:34.7635118Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T03:59:34.7644223Z 
2022-05-18T03:59:34.7644354Z Running tests...
2022-05-18T03:59:34.7644946Z ----------------------------------------------------------------------
2022-05-18T03:59:35.0760222Z   test_async_function_multi_fanout_async (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9515
2022-05-18T03:59:35.0783402Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9516
2022-05-18T03:59:35.0806621Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9517
2022-05-18T03:59:35.0831167Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9518
2022-05-18T03:59:35.6921926Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa1mq7c78
2022-05-18T03:59:35.6925120Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa1mq7c78/_remote_module_non_scriptable.py
2022-05-18T03:59:35.7071095Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2n669r7m
2022-05-18T03:59:35.7072117Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2n669r7m/_remote_module_non_scriptable.py
2022-05-18T03:59:35.7242691Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7i0sxqie
2022-05-18T03:59:35.7244204Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7i0sxqie/_remote_module_non_scriptable.py
2022-05-18T03:59:35.7413817Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpriqh_qba
2022-05-18T03:59:35.7414907Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpriqh_qba/_remote_module_non_scriptable.py
2022-05-18T03:59:35.9415926Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:59:35.9570058Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:59:35.9708201Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:59:35.9885492Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:59:36.4871232Z ok (1.722s)
2022-05-18T03:59:36.4871436Z 
2022-05-18T03:59:36.4871908Z ----------------------------------------------------------------------
2022-05-18T03:59:36.4872370Z Ran 1 test in 1.723s
2022-05-18T03:59:36.4872598Z 
2022-05-18T03:59:36.4872692Z OK
2022-05-18T03:59:36.4872829Z 
2022-05-18T03:59:36.4872925Z Generating XML reports...
2022-05-18T03:59:36.4905744Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518035934.xml
2022-05-18T03:59:37.2533165Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkm7gv53k
2022-05-18T03:59:37.2533971Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkm7gv53k/_remote_module_non_scriptable.py
2022-05-18T03:59:37.5085780Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T03:59:37.5096142Z 
2022-05-18T03:59:37.5096573Z Running tests...
2022-05-18T03:59:37.5096997Z ----------------------------------------------------------------------
2022-05-18T03:59:37.8225338Z   test_async_function_multi_fanout_remote (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9734
2022-05-18T03:59:37.8248366Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9735
2022-05-18T03:59:37.8271450Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9736
2022-05-18T03:59:37.8295464Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9737
2022-05-18T03:59:38.4591244Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_q3_4wj6
2022-05-18T03:59:38.4592011Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_q3_4wj6/_remote_module_non_scriptable.py
2022-05-18T03:59:38.4625683Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8wu5llbb
2022-05-18T03:59:38.4627543Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8wu5llbb/_remote_module_non_scriptable.py
2022-05-18T03:59:38.4632771Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgdo_ddb3
2022-05-18T03:59:38.4634976Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgdo_ddb3/_remote_module_non_scriptable.py
2022-05-18T03:59:38.4648683Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3l_e81f4
2022-05-18T03:59:38.4650094Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3l_e81f4/_remote_module_non_scriptable.py
2022-05-18T03:59:38.7067170Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:59:38.7099227Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:59:38.7106158Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:59:38.7155894Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:59:39.1334601Z ok (1.623s)
2022-05-18T03:59:39.1334833Z 
2022-05-18T03:59:39.1335287Z ----------------------------------------------------------------------
2022-05-18T03:59:39.1335709Z Ran 1 test in 1.624s
2022-05-18T03:59:39.1335903Z 
2022-05-18T03:59:39.1335983Z OK
2022-05-18T03:59:39.1336128Z 
2022-05-18T03:59:39.1336270Z Generating XML reports...
2022-05-18T03:59:39.1369664Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518035937.xml
2022-05-18T03:59:39.9034916Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcsitgrpe
2022-05-18T03:59:39.9035883Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcsitgrpe/_remote_module_non_scriptable.py
2022-05-18T03:59:40.1587264Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T03:59:40.1597452Z 
2022-05-18T03:59:40.1598036Z Running tests...
2022-05-18T03:59:40.1598614Z ----------------------------------------------------------------------
2022-05-18T03:59:40.4778107Z   test_async_function_nested (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9953
2022-05-18T03:59:40.4800825Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9954
2022-05-18T03:59:40.4824681Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9955
2022-05-18T03:59:40.4848679Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9956
2022-05-18T03:59:41.1310221Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt17uegzc
2022-05-18T03:59:41.1311292Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt17uegzc/_remote_module_non_scriptable.py
2022-05-18T03:59:41.1390905Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2tva_jsk
2022-05-18T03:59:41.1391989Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2tva_jsk/_remote_module_non_scriptable.py
2022-05-18T03:59:41.1958776Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo8co30hm
2022-05-18T03:59:41.1960058Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo8co30hm/_remote_module_non_scriptable.py
2022-05-18T03:59:41.2074656Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8vlje2uk
2022-05-18T03:59:41.2075698Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8vlje2uk/_remote_module_non_scriptable.py
2022-05-18T03:59:41.3811733Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:59:41.3868452Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:59:41.4443960Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:59:41.4521231Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:59:41.9891107Z ok (1.829s)
2022-05-18T03:59:41.9891391Z 
2022-05-18T03:59:41.9891892Z ----------------------------------------------------------------------
2022-05-18T03:59:41.9892156Z Ran 1 test in 1.829s
2022-05-18T03:59:41.9892290Z 
2022-05-18T03:59:41.9892353Z OK
2022-05-18T03:59:41.9892446Z 
2022-05-18T03:59:41.9892544Z Generating XML reports...
2022-05-18T03:59:41.9926168Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518035940.xml
2022-05-18T03:59:42.7504858Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo8hb_jf1
2022-05-18T03:59:42.7505791Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo8hb_jf1/_remote_module_non_scriptable.py
2022-05-18T03:59:43.0007510Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T03:59:43.0017599Z 
2022-05-18T03:59:43.0017706Z Running tests...
2022-05-18T03:59:43.0018319Z ----------------------------------------------------------------------
2022-05-18T03:59:43.3147935Z   test_async_function_nested_remote (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10172
2022-05-18T03:59:43.3169766Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10173
2022-05-18T03:59:43.3193378Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10174
2022-05-18T03:59:43.3217131Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10175
2022-05-18T03:59:43.9307347Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcxawj2zv
2022-05-18T03:59:43.9308761Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcxawj2zv/_remote_module_non_scriptable.py
2022-05-18T03:59:43.9432134Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx_eb5d6_
2022-05-18T03:59:43.9434127Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx_eb5d6_/_remote_module_non_scriptable.py
2022-05-18T03:59:43.9434817Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7mb_q9yk
2022-05-18T03:59:43.9436688Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7mb_q9yk/_remote_module_non_scriptable.py
2022-05-18T03:59:43.9574810Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwikoavsx
2022-05-18T03:59:43.9576047Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwikoavsx/_remote_module_non_scriptable.py
2022-05-18T03:59:44.1786005Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:59:44.1903065Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:59:44.1908772Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:59:44.2045908Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:59:44.7256865Z ok (1.724s)
2022-05-18T03:59:44.7257115Z 
2022-05-18T03:59:44.7257666Z ----------------------------------------------------------------------
2022-05-18T03:59:44.7257977Z Ran 1 test in 1.724s
2022-05-18T03:59:44.7258325Z 
2022-05-18T03:59:44.7258393Z OK
2022-05-18T03:59:44.7258486Z 
2022-05-18T03:59:44.7258580Z Generating XML reports...
2022-05-18T03:59:44.7291470Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518035942.xml
2022-05-18T03:59:45.4864986Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxvra27li
2022-05-18T03:59:45.4866041Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxvra27li/_remote_module_non_scriptable.py
2022-05-18T03:59:45.7387287Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T03:59:45.7396816Z 
2022-05-18T03:59:45.7397170Z Running tests...
2022-05-18T03:59:45.7397595Z ----------------------------------------------------------------------
2022-05-18T03:59:46.0571495Z   test_async_function_raise (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10391
2022-05-18T03:59:46.0593887Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10392
2022-05-18T03:59:46.0617169Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10393
2022-05-18T03:59:46.0641163Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10394
2022-05-18T03:59:46.6658234Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph7oofntq
2022-05-18T03:59:46.6659033Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph7oofntq/_remote_module_non_scriptable.py
2022-05-18T03:59:46.6847912Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsvqj3ooz
2022-05-18T03:59:46.6848694Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsvqj3ooz/_remote_module_non_scriptable.py
2022-05-18T03:59:46.6876228Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphm9ur58g
2022-05-18T03:59:46.6877896Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphm9ur58g/_remote_module_non_scriptable.py
2022-05-18T03:59:46.7090518Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv6hwmc3d
2022-05-18T03:59:46.7091705Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv6hwmc3d/_remote_module_non_scriptable.py
2022-05-18T03:59:46.9136926Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:59:46.9348529Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:59:46.9361881Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:59:46.9547183Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:59:47.1491942Z On WorkerInfo(id=1, name=worker1):
2022-05-18T03:59:47.1492683Z RuntimeError('Expected error')
2022-05-18T03:59:47.1493153Z Traceback (most recent call last):
2022-05-18T03:59:47.1494140Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function
2022-05-18T03:59:47.1494851Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-05-18T03:59:47.1495757Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/functions.py", line 162, in wrapper
2022-05-18T03:59:47.1496360Z     return fn(*args, **kwargs)
2022-05-18T03:59:47.1497266Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 468, in async_raise_func
2022-05-18T03:59:47.1497937Z     raise RuntimeError("Expected error")
2022-05-18T03:59:47.1498407Z RuntimeError: Expected error
2022-05-18T03:59:47.1498683Z 
2022-05-18T03:59:47.1691493Z On WorkerInfo(id=0, name=worker0):
2022-05-18T03:59:47.1692065Z RuntimeError('Expected error')
2022-05-18T03:59:47.1692751Z Traceback (most recent call last):
2022-05-18T03:59:47.1693539Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function
2022-05-18T03:59:47.1694242Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-05-18T03:59:47.1704246Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/functions.py", line 162, in wrapper
2022-05-18T03:59:47.1704751Z     return fn(*args, **kwargs)
2022-05-18T03:59:47.1705476Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 468, in async_raise_func
2022-05-18T03:59:47.1706034Z     raise RuntimeError("Expected error")
2022-05-18T03:59:47.1706410Z RuntimeError: Expected error
2022-05-18T03:59:47.1706642Z 
2022-05-18T03:59:47.1782212Z On WorkerInfo(id=3, name=worker3):
2022-05-18T03:59:47.1782754Z RuntimeError('Expected error')
2022-05-18T03:59:47.1783208Z Traceback (most recent call last):
2022-05-18T03:59:47.1783970Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function
2022-05-18T03:59:47.1784581Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-05-18T03:59:47.1785349Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/functions.py", line 162, in wrapper
2022-05-18T03:59:47.1785835Z     return fn(*args, **kwargs)
2022-05-18T03:59:47.1786539Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 468, in async_raise_func
2022-05-18T03:59:47.1787020Z     raise RuntimeError("Expected error")
2022-05-18T03:59:47.1787245Z RuntimeError: Expected error
2022-05-18T03:59:47.1787430Z 
2022-05-18T03:59:47.1808787Z On WorkerInfo(id=2, name=worker2):
2022-05-18T03:59:47.1809469Z RuntimeError('Expected error')
2022-05-18T03:59:47.1809936Z Traceback (most recent call last):
2022-05-18T03:59:47.1810818Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function
2022-05-18T03:59:47.1811528Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-05-18T03:59:47.1812431Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/functions.py", line 162, in wrapper
2022-05-18T03:59:47.1813034Z     return fn(*args, **kwargs)
2022-05-18T03:59:47.1813977Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 468, in async_raise_func
2022-05-18T03:59:47.1814671Z     raise RuntimeError("Expected error")
2022-05-18T03:59:47.1815139Z RuntimeError: Expected error
2022-05-18T03:59:47.1815412Z 
2022-05-18T03:59:47.3678871Z ok (1.628s)
2022-05-18T03:59:47.3679090Z 
2022-05-18T03:59:47.3679512Z ----------------------------------------------------------------------
2022-05-18T03:59:47.3679857Z Ran 1 test in 1.628s
2022-05-18T03:59:47.3680013Z 
2022-05-18T03:59:47.3680084Z OK
2022-05-18T03:59:47.3680180Z 
2022-05-18T03:59:47.3680277Z Generating XML reports...
2022-05-18T03:59:47.3714633Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518035945.xml
2022-05-18T03:59:48.1346595Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjvo3agzx
2022-05-18T03:59:48.1347087Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjvo3agzx/_remote_module_non_scriptable.py
2022-05-18T03:59:48.3875670Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T03:59:48.3885411Z 
2022-05-18T03:59:48.3885504Z Running tests...
2022-05-18T03:59:48.3885904Z ----------------------------------------------------------------------
2022-05-18T03:59:48.6917658Z   test_async_function_raise_async (__main__.TensorPipeRpcTest) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/76907 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure IN_CI is not set and you are not using the flag --import-disabled-tests. (0.303s)
2022-05-18T03:59:48.6918407Z 
2022-05-18T03:59:48.6918612Z ----------------------------------------------------------------------
2022-05-18T03:59:48.6918940Z Ran 1 test in 0.303s
2022-05-18T03:59:48.6919057Z 
2022-05-18T03:59:48.6919131Z OK (skipped=1)
2022-05-18T03:59:48.6919238Z 
2022-05-18T03:59:48.6919324Z Generating XML reports...
2022-05-18T03:59:48.6940591Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518035948.xml
2022-05-18T03:59:49.4000067Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpstfrk35s
2022-05-18T03:59:49.4000569Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpstfrk35s/_remote_module_non_scriptable.py
2022-05-18T03:59:49.6517976Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T03:59:49.6527870Z 
2022-05-18T03:59:49.6527967Z Running tests...
2022-05-18T03:59:49.6528802Z ----------------------------------------------------------------------
2022-05-18T03:59:49.9628925Z   test_async_function_raise_remote (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10620
2022-05-18T03:59:49.9650736Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10621
2022-05-18T03:59:49.9674719Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10622
2022-05-18T03:59:49.9699903Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10623
2022-05-18T03:59:50.5922724Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp32myel_a
2022-05-18T03:59:50.5923521Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp32myel_a/_remote_module_non_scriptable.py
2022-05-18T03:59:50.6043045Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp70c78gvl
2022-05-18T03:59:50.6044049Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp70c78gvl/_remote_module_non_scriptable.py
2022-05-18T03:59:50.6616696Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptfs6nqts
2022-05-18T03:59:50.6617531Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptfs6nqts/_remote_module_non_scriptable.py
2022-05-18T03:59:50.6665281Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp526mlq3e
2022-05-18T03:59:50.6666588Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp526mlq3e/_remote_module_non_scriptable.py
2022-05-18T03:59:50.8418005Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:59:50.8500429Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:59:50.9069558Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:59:50.9133524Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:59:51.1132413Z On WorkerInfo(id=1, name=worker1):
2022-05-18T03:59:51.1133115Z RuntimeError('Expected error')
2022-05-18T03:59:51.1133671Z Traceback (most recent call last):
2022-05-18T03:59:51.1134574Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function
2022-05-18T03:59:51.1135289Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-05-18T03:59:51.1136205Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/functions.py", line 162, in wrapper
2022-05-18T03:59:51.1136805Z     return fn(*args, **kwargs)
2022-05-18T03:59:51.1137687Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 468, in async_raise_func
2022-05-18T03:59:51.1138369Z     raise RuntimeError("Expected error")
2022-05-18T03:59:51.1138848Z RuntimeError: Expected error
2022-05-18T03:59:51.1139118Z 
2022-05-18T03:59:51.1330578Z On WorkerInfo(id=0, name=worker0):
2022-05-18T03:59:51.1331180Z RuntimeError('Expected error')
2022-05-18T03:59:51.1331566Z Traceback (most recent call last):
2022-05-18T03:59:51.1332515Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function
2022-05-18T03:59:51.1333079Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-05-18T03:59:51.1333872Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/functions.py", line 162, in wrapper
2022-05-18T03:59:51.1334374Z     return fn(*args, **kwargs)
2022-05-18T03:59:51.1335073Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 468, in async_raise_func
2022-05-18T03:59:51.1335610Z     raise RuntimeError("Expected error")
2022-05-18T03:59:51.1335999Z RuntimeError: Expected error
2022-05-18T03:59:51.1336216Z 
2022-05-18T03:59:51.1416585Z On WorkerInfo(id=3, name=worker3):
2022-05-18T03:59:51.1417145Z RuntimeError('Expected error')
2022-05-18T03:59:51.1420164Z Traceback (most recent call last):
2022-05-18T03:59:51.1421197Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function
2022-05-18T03:59:51.1421951Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-05-18T03:59:51.1422946Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/functions.py", line 162, in wrapper
2022-05-18T03:59:51.1423545Z     return fn(*args, **kwargs)
2022-05-18T03:59:51.1424435Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 468, in async_raise_func
2022-05-18T03:59:51.1425134Z     raise RuntimeError("Expected error")
2022-05-18T03:59:51.1425603Z RuntimeError: Expected error
2022-05-18T03:59:51.1425872Z 
2022-05-18T03:59:51.1454103Z On WorkerInfo(id=2, name=worker2):
2022-05-18T03:59:51.1454540Z RuntimeError('Expected error')
2022-05-18T03:59:51.1454891Z Traceback (most recent call last):
2022-05-18T03:59:51.1455666Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function
2022-05-18T03:59:51.1456380Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-05-18T03:59:51.1457284Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/functions.py", line 162, in wrapper
2022-05-18T03:59:51.1457887Z     return fn(*args, **kwargs)
2022-05-18T03:59:51.1458756Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 468, in async_raise_func
2022-05-18T03:59:51.1459450Z     raise RuntimeError("Expected error")
2022-05-18T03:59:51.1459919Z RuntimeError: Expected error
2022-05-18T03:59:51.1460190Z 
2022-05-18T03:59:51.3738615Z ok (1.721s)
2022-05-18T03:59:51.3738802Z 
2022-05-18T03:59:51.3739130Z ----------------------------------------------------------------------
2022-05-18T03:59:51.3739436Z Ran 1 test in 1.721s
2022-05-18T03:59:51.3739553Z 
2022-05-18T03:59:51.3739618Z OK
2022-05-18T03:59:51.3739731Z 
2022-05-18T03:59:51.3739812Z Generating XML reports...
2022-05-18T03:59:51.3772979Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518035949.xml
2022-05-18T03:59:52.1429733Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6b7el6h3
2022-05-18T03:59:52.1430479Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6b7el6h3/_remote_module_non_scriptable.py
2022-05-18T03:59:52.3994425Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T03:59:52.4002963Z 
2022-05-18T03:59:52.4003061Z Running tests...
2022-05-18T03:59:52.4003872Z ----------------------------------------------------------------------
2022-05-18T03:59:52.7160043Z   test_async_function_simple (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10839
2022-05-18T03:59:52.7183483Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10840
2022-05-18T03:59:52.7206391Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10841
2022-05-18T03:59:52.7231378Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10842
2022-05-18T03:59:53.3759706Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7juwl6gx
2022-05-18T03:59:53.3760960Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7juwl6gx/_remote_module_non_scriptable.py
2022-05-18T03:59:53.4002290Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu3b7fwwi
2022-05-18T03:59:53.4003049Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu3b7fwwi/_remote_module_non_scriptable.py
2022-05-18T03:59:53.4384806Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn5f3auv2
2022-05-18T03:59:53.4385553Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn5f3auv2/_remote_module_non_scriptable.py
2022-05-18T03:59:53.4450266Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnt7xpswj
2022-05-18T03:59:53.4451661Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnt7xpswj/_remote_module_non_scriptable.py
2022-05-18T03:59:53.6242183Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:59:53.6454591Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:59:53.6849489Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:59:53.6934973Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:59:54.1272039Z ok (1.727s)
2022-05-18T03:59:54.1272177Z 
2022-05-18T03:59:54.1272485Z ----------------------------------------------------------------------
2022-05-18T03:59:54.1272732Z Ran 1 test in 1.727s
2022-05-18T03:59:54.1272856Z 
2022-05-18T03:59:54.1272918Z OK
2022-05-18T03:59:54.1273025Z 
2022-05-18T03:59:54.1273110Z Generating XML reports...
2022-05-18T03:59:54.1306924Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518035952.xml
2022-05-18T03:59:54.8935988Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpaahxwbmt
2022-05-18T03:59:54.8936885Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpaahxwbmt/_remote_module_non_scriptable.py
2022-05-18T03:59:55.1458062Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T03:59:55.1467815Z 
2022-05-18T03:59:55.1467948Z Running tests...
2022-05-18T03:59:55.1468559Z ----------------------------------------------------------------------
2022-05-18T03:59:55.4623538Z   test_async_function_with_future_ctor (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11058
2022-05-18T03:59:55.4645413Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11059
2022-05-18T03:59:55.4668455Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11060
2022-05-18T03:59:55.4692748Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11061
2022-05-18T03:59:56.1411871Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgsphit94
2022-05-18T03:59:56.1412630Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgsphit94/_remote_module_non_scriptable.py
2022-05-18T03:59:56.1674166Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpauyi7ql5
2022-05-18T03:59:56.1674958Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpauyi7ql5/_remote_module_non_scriptable.py
2022-05-18T03:59:56.1925292Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdgypte_y
2022-05-18T03:59:56.1926241Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdgypte_y/_remote_module_non_scriptable.py
2022-05-18T03:59:56.2122083Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp19cagz1r
2022-05-18T03:59:56.2123345Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp19cagz1r/_remote_module_non_scriptable.py
2022-05-18T03:59:56.3905491Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:59:56.4151076Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:59:56.4416771Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:59:56.4583072Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:59:56.9733050Z ok (1.826s)
2022-05-18T03:59:56.9733442Z 
2022-05-18T03:59:56.9734067Z ----------------------------------------------------------------------
2022-05-18T03:59:56.9734497Z Ran 1 test in 1.826s
2022-05-18T03:59:56.9734681Z 
2022-05-18T03:59:56.9734801Z OK
2022-05-18T03:59:56.9734946Z 
2022-05-18T03:59:56.9735082Z Generating XML reports...
2022-05-18T03:59:56.9769482Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518035955.xml
2022-05-18T03:59:57.7473634Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg6_e7fsu
2022-05-18T03:59:57.7474427Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg6_e7fsu/_remote_module_non_scriptable.py
2022-05-18T03:59:58.0006354Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T03:59:58.0015968Z 
2022-05-18T03:59:58.0016116Z Running tests...
2022-05-18T03:59:58.0016559Z ----------------------------------------------------------------------
2022-05-18T03:59:58.3214105Z   test_async_function_with_future_ctor_remote (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11277
2022-05-18T03:59:58.3237161Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11278
2022-05-18T03:59:58.3261344Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11279
2022-05-18T03:59:58.3285430Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11280
2022-05-18T03:59:58.9440688Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcb7wroac
2022-05-18T03:59:58.9441452Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcb7wroac/_remote_module_non_scriptable.py
2022-05-18T03:59:58.9549353Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgmhj99j2
2022-05-18T03:59:58.9550555Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgmhj99j2/_remote_module_non_scriptable.py
2022-05-18T03:59:58.9580019Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu_oni6bi
2022-05-18T03:59:58.9582150Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu_oni6bi/_remote_module_non_scriptable.py
2022-05-18T03:59:58.9646115Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplzfbjfe8
2022-05-18T03:59:58.9648012Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplzfbjfe8/_remote_module_non_scriptable.py
2022-05-18T03:59:59.1912652Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T03:59:59.2009223Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T03:59:59.2064748Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T03:59:59.2130413Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T03:59:59.7324676Z ok (1.731s)
2022-05-18T03:59:59.7324926Z 
2022-05-18T03:59:59.7325561Z ----------------------------------------------------------------------
2022-05-18T03:59:59.7325918Z Ran 1 test in 1.731s
2022-05-18T03:59:59.7326035Z 
2022-05-18T03:59:59.7326350Z OK
2022-05-18T03:59:59.7326441Z 
2022-05-18T03:59:59.7326535Z Generating XML reports...
2022-05-18T03:59:59.7360503Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518035957.xml
2022-05-18T04:00:00.5018259Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpiw9uasjv
2022-05-18T04:00:00.5018992Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpiw9uasjv/_remote_module_non_scriptable.py
2022-05-18T04:00:00.7527546Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:00:00.7537541Z 
2022-05-18T04:00:00.7537634Z Running tests...
2022-05-18T04:00:00.7538413Z ----------------------------------------------------------------------
2022-05-18T04:00:01.0654092Z   test_async_function_wrong_return_type (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11496
2022-05-18T04:00:01.0676369Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11497
2022-05-18T04:00:01.0699367Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11498
2022-05-18T04:00:01.0723288Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11499
2022-05-18T04:00:01.7036295Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp88iv4ahf
2022-05-18T04:00:01.7037050Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp88iv4ahf/_remote_module_non_scriptable.py
2022-05-18T04:00:01.7051005Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_e6srets
2022-05-18T04:00:01.7052285Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_e6srets/_remote_module_non_scriptable.py
2022-05-18T04:00:01.7130585Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6s0ax93w
2022-05-18T04:00:01.7131529Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6s0ax93w/_remote_module_non_scriptable.py
2022-05-18T04:00:01.7191282Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpehwc98zz
2022-05-18T04:00:01.7192253Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpehwc98zz/_remote_module_non_scriptable.py
2022-05-18T04:00:01.9593615Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:00:01.9620239Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:00:01.9681790Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:00:01.9733639Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:00:02.4763461Z ok (1.722s)
2022-05-18T04:00:02.4763679Z 
2022-05-18T04:00:02.4764291Z ----------------------------------------------------------------------
2022-05-18T04:00:02.4764719Z Ran 1 test in 1.723s
2022-05-18T04:00:02.4764860Z 
2022-05-18T04:00:02.4764936Z OK
2022-05-18T04:00:02.4765029Z 
2022-05-18T04:00:02.4765106Z Generating XML reports...
2022-05-18T04:00:02.4799939Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040000.xml
2022-05-18T04:00:03.2501859Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy40h525k
2022-05-18T04:00:03.2502815Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy40h525k/_remote_module_non_scriptable.py
2022-05-18T04:00:03.5058812Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:00:03.5068711Z 
2022-05-18T04:00:03.5068828Z Running tests...
2022-05-18T04:00:03.5069713Z ----------------------------------------------------------------------
2022-05-18T04:00:03.8271942Z   test_async_function_wrong_return_type_async (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11715
2022-05-18T04:00:03.8294362Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11716
2022-05-18T04:00:03.8318068Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11717
2022-05-18T04:00:03.8342421Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11718
2022-05-18T04:00:04.4616914Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpopgnpu_4
2022-05-18T04:00:04.4617716Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpopgnpu_4/_remote_module_non_scriptable.py
2022-05-18T04:00:04.4746463Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbc8j8p84
2022-05-18T04:00:04.4747211Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp767503mk
2022-05-18T04:00:04.4748335Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbc8j8p84/_remote_module_non_scriptable.py
2022-05-18T04:00:04.4749058Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp767503mk/_remote_module_non_scriptable.py
2022-05-18T04:00:04.4778135Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn0x85yai
2022-05-18T04:00:04.4779697Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn0x85yai/_remote_module_non_scriptable.py
2022-05-18T04:00:04.7110816Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:00:04.7235789Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:00:04.7253127Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:00:04.7263606Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:00:05.2383740Z ok (1.731s)
2022-05-18T04:00:05.2384162Z 
2022-05-18T04:00:05.2384625Z ----------------------------------------------------------------------
2022-05-18T04:00:05.2385009Z Ran 1 test in 1.731s
2022-05-18T04:00:05.2385206Z 
2022-05-18T04:00:05.2385305Z OK
2022-05-18T04:00:05.2385437Z 
2022-05-18T04:00:05.2385578Z Generating XML reports...
2022-05-18T04:00:05.2419616Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040003.xml
2022-05-18T04:00:06.0251670Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe4lgmqz9
2022-05-18T04:00:06.0252463Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe4lgmqz9/_remote_module_non_scriptable.py
2022-05-18T04:00:06.2797650Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:00:06.2807597Z 
2022-05-18T04:00:06.2807731Z Running tests...
2022-05-18T04:00:06.2808122Z ----------------------------------------------------------------------
2022-05-18T04:00:06.5984049Z   test_async_function_wrong_return_type_remote (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11934
2022-05-18T04:00:06.6006348Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11935
2022-05-18T04:00:06.6029705Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11936
2022-05-18T04:00:06.6054954Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11937
2022-05-18T04:00:07.1903163Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0zcgz9vd
2022-05-18T04:00:07.1903959Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0zcgz9vd/_remote_module_non_scriptable.py
2022-05-18T04:00:07.2333579Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw26mpm0p
2022-05-18T04:00:07.2335160Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw26mpm0p/_remote_module_non_scriptable.py
2022-05-18T04:00:07.2600237Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptk6k54rq
2022-05-18T04:00:07.2600975Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptk6k54rq/_remote_module_non_scriptable.py
2022-05-18T04:00:07.2706093Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4d4t9_gy
2022-05-18T04:00:07.2707510Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4d4t9_gy/_remote_module_non_scriptable.py
2022-05-18T04:00:07.4408359Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:00:07.4819649Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:00:07.5098708Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:00:07.5220735Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:00:08.0093559Z ok (1.728s)
2022-05-18T04:00:08.0093776Z 
2022-05-18T04:00:08.0094505Z ----------------------------------------------------------------------
2022-05-18T04:00:08.0094935Z Ran 1 test in 1.729s
2022-05-18T04:00:08.0095160Z 
2022-05-18T04:00:08.0095238Z OK
2022-05-18T04:00:08.0095385Z 
2022-05-18T04:00:08.0095516Z Generating XML reports...
2022-05-18T04:00:08.0129800Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040006.xml
2022-05-18T04:00:08.7757791Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphsvr_we0
2022-05-18T04:00:08.7758548Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphsvr_we0/_remote_module_non_scriptable.py
2022-05-18T04:00:09.0276096Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:00:09.0285268Z 
2022-05-18T04:00:09.0285365Z Running tests...
2022-05-18T04:00:09.0285783Z ----------------------------------------------------------------------
2022-05-18T04:00:09.3402351Z   test_async_record_function_cbs_jit_call (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12153
2022-05-18T04:00:09.3425100Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12154
2022-05-18T04:00:09.3448830Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12155
2022-05-18T04:00:09.3472955Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 12156
2022-05-18T04:00:09.9451809Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppjtmlg11
2022-05-18T04:00:09.9452548Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppjtmlg11/_remote_module_non_scriptable.py
2022-05-18T04:00:09.9731638Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp93aw1l8s
2022-05-18T04:00:09.9732513Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp93aw1l8s/_remote_module_non_scriptable.py
2022-05-18T04:00:10.0122275Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2rm586qb
2022-05-18T04:00:10.0123466Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2rm586qb/_remote_module_non_scriptable.py
2022-05-18T04:00:10.0168991Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy_hzuiir
2022-05-18T04:00:10.0170520Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy_hzuiir/_remote_module_non_scriptable.py
2022-05-18T04:00:10.1947353Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:00:10.2202906Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:00:10.2604816Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:00:10.2673220Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:00:10.7513562Z ok (1.723s)
2022-05-18T04:00:10.7513799Z 
2022-05-18T04:00:10.7514186Z ----------------------------------------------------------------------
2022-05-18T04:00:10.7514477Z Ran 1 test in 1.723s
2022-05-18T04:00:10.7514861Z 
2022-05-18T04:00:10.7514925Z OK
2022-05-18T04:00:10.7515071Z 
2022-05-18T04:00:10.7515159Z Generating XML reports...
2022-05-18T04:00:10.7548244Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040009.xml
2022-05-18T04:00:11.5142174Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnvz1acws
2022-05-18T04:00:11.5143129Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnvz1acws/_remote_module_non_scriptable.py
2022-05-18T04:00:11.7656462Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:00:11.7666389Z 
2022-05-18T04:00:11.7666807Z Running tests...
2022-05-18T04:00:11.7667225Z ----------------------------------------------------------------------
2022-05-18T04:00:12.0777792Z   test_async_record_function_double_end_callbacks (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12372
2022-05-18T04:00:12.0800828Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12373
2022-05-18T04:00:12.0824650Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12374
2022-05-18T04:00:12.0848347Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 12375
2022-05-18T04:00:12.6872167Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkmrxvv9z
2022-05-18T04:00:12.6872917Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkmrxvv9z/_remote_module_non_scriptable.py
2022-05-18T04:00:12.7078107Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw3tk_xpw
2022-05-18T04:00:12.7079316Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw3tk_xpw/_remote_module_non_scriptable.py
2022-05-18T04:00:12.7156227Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfx6tq3q9
2022-05-18T04:00:12.7157857Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfx6tq3q9/_remote_module_non_scriptable.py
2022-05-18T04:00:12.7414071Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpplk0qes4
2022-05-18T04:00:12.7415149Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpplk0qes4/_remote_module_non_scriptable.py
2022-05-18T04:00:12.9334204Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:00:12.9552460Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:00:12.9627584Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:00:12.9884916Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:00:14.4904835Z ok (2.724s)
2022-05-18T04:00:14.4905085Z 
2022-05-18T04:00:14.4905521Z ----------------------------------------------------------------------
2022-05-18T04:00:14.4905900Z Ran 1 test in 2.724s
2022-05-18T04:00:14.4906039Z 
2022-05-18T04:00:14.4906105Z OK
2022-05-18T04:00:14.4906184Z 
2022-05-18T04:00:14.4906283Z Generating XML reports...
2022-05-18T04:00:14.4942025Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040011.xml
2022-05-18T04:00:15.2567953Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnodxfmw6
2022-05-18T04:00:15.2568798Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnodxfmw6/_remote_module_non_scriptable.py
2022-05-18T04:00:15.5100402Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:00:15.5110058Z 
2022-05-18T04:00:15.5110194Z Running tests...
2022-05-18T04:00:15.5110612Z ----------------------------------------------------------------------
2022-05-18T04:00:15.8240607Z   test_async_record_function_double_end_callbacks_new_signatures (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12591
2022-05-18T04:00:15.8264104Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12592
2022-05-18T04:00:15.8287175Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12593
2022-05-18T04:00:15.8312336Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 12594
2022-05-18T04:00:16.5214857Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1fs0hafl
2022-05-18T04:00:16.5215635Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1fs0hafl/_remote_module_non_scriptable.py
2022-05-18T04:00:16.5258700Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp87h4un2_
2022-05-18T04:00:16.5260051Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp87h4un2_/_remote_module_non_scriptable.py
2022-05-18T04:00:16.5348705Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq7ltan07
2022-05-18T04:00:16.5350077Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq7ltan07/_remote_module_non_scriptable.py
2022-05-18T04:00:16.5509055Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbbhir8h5
2022-05-18T04:00:16.5510330Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbbhir8h5/_remote_module_non_scriptable.py
2022-05-18T04:00:16.7717743Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:00:16.7754425Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:00:16.7824879Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:00:16.7982468Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:00:18.2367793Z ok (2.725s)
2022-05-18T04:00:18.2368233Z 
2022-05-18T04:00:18.2369135Z ----------------------------------------------------------------------
2022-05-18T04:00:18.2369630Z Ran 1 test in 2.726s
2022-05-18T04:00:18.2369764Z 
2022-05-18T04:00:18.2369811Z OK
2022-05-18T04:00:18.2370238Z 
2022-05-18T04:00:18.2370440Z Generating XML reports...
2022-05-18T04:00:18.2403198Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040015.xml
2022-05-18T04:00:19.0077677Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeupnz249
2022-05-18T04:00:19.0078447Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeupnz249/_remote_module_non_scriptable.py
2022-05-18T04:00:19.2606457Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:00:19.2615244Z 
2022-05-18T04:00:19.2615336Z Running tests...
2022-05-18T04:00:19.2616369Z ----------------------------------------------------------------------
2022-05-18T04:00:19.5744260Z   test_async_static_method (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12810
2022-05-18T04:00:19.5765528Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12811
2022-05-18T04:00:19.5788459Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12812
2022-05-18T04:00:19.5813259Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 12813
2022-05-18T04:00:20.2372260Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvn9u6ggr
2022-05-18T04:00:20.2373006Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvn9u6ggr/_remote_module_non_scriptable.py
2022-05-18T04:00:20.2418084Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzieafzwn
2022-05-18T04:00:20.2419630Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzieafzwn/_remote_module_non_scriptable.py
2022-05-18T04:00:20.2715841Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7e9i53u4
2022-05-18T04:00:20.2717331Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7e9i53u4/_remote_module_non_scriptable.py
2022-05-18T04:00:20.2743422Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpelhrswb7
2022-05-18T04:00:20.2745696Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpelhrswb7/_remote_module_non_scriptable.py
2022-05-18T04:00:20.4838871Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:00:20.4892464Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:00:20.5197687Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:00:20.5215347Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:00:20.9851868Z ok (1.723s)
2022-05-18T04:00:20.9852081Z 
2022-05-18T04:00:20.9853479Z ----------------------------------------------------------------------
2022-05-18T04:00:20.9853789Z Ran 1 test in 1.724s
2022-05-18T04:00:20.9853907Z 
2022-05-18T04:00:20.9853968Z OK
2022-05-18T04:00:20.9854060Z 
2022-05-18T04:00:20.9854140Z Generating XML reports...
2022-05-18T04:00:20.9887635Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040019.xml
2022-05-18T04:00:21.7597885Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpexttpz8e
2022-05-18T04:00:21.7599072Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpexttpz8e/_remote_module_non_scriptable.py
2022-05-18T04:00:22.0107910Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:00:22.0117159Z 
2022-05-18T04:00:22.0117250Z Running tests...
2022-05-18T04:00:22.0117686Z ----------------------------------------------------------------------
2022-05-18T04:00:22.3331307Z   test_async_static_method_remote (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13029
2022-05-18T04:00:22.3356962Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13030
2022-05-18T04:00:22.3380411Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13031
2022-05-18T04:00:22.3405286Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 13032
2022-05-18T04:00:22.9158734Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxtvpc5dd
2022-05-18T04:00:22.9159953Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxtvpc5dd/_remote_module_non_scriptable.py
2022-05-18T04:00:22.9186680Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsazpyswc
2022-05-18T04:00:22.9189410Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsazpyswc/_remote_module_non_scriptable.py
2022-05-18T04:00:22.9689269Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdlwdpwae
2022-05-18T04:00:22.9690332Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdlwdpwae/_remote_module_non_scriptable.py
2022-05-18T04:00:22.9731403Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph4x9ck3c
2022-05-18T04:00:22.9733468Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph4x9ck3c/_remote_module_non_scriptable.py
2022-05-18T04:00:23.1643003Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:00:23.1657029Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:00:23.2153643Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:00:23.2224740Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:00:23.6442412Z ok (1.632s)
2022-05-18T04:00:23.6442572Z 
2022-05-18T04:00:23.6442894Z ----------------------------------------------------------------------
2022-05-18T04:00:23.6443883Z Ran 1 test in 1.632s
2022-05-18T04:00:23.6444115Z 
2022-05-18T04:00:23.6444205Z OK
2022-05-18T04:00:23.6444346Z 
2022-05-18T04:00:23.6444493Z Generating XML reports...
2022-05-18T04:00:23.6479338Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040022.xml
2022-05-18T04:00:24.4314073Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv6cpfq3d
2022-05-18T04:00:24.4314889Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv6cpfq3d/_remote_module_non_scriptable.py
2022-05-18T04:00:24.6866457Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:00:24.6875671Z 
2022-05-18T04:00:24.6875765Z Running tests...
2022-05-18T04:00:24.6876220Z ----------------------------------------------------------------------
2022-05-18T04:00:25.0027022Z   test_build_rpc_profiling_key (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13248
2022-05-18T04:00:25.0050108Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13249
2022-05-18T04:00:25.0074075Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13250
2022-05-18T04:00:25.0098235Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 13251
2022-05-18T04:00:25.6697007Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp434doai0
2022-05-18T04:00:25.6697737Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp434doai0/_remote_module_non_scriptable.py
2022-05-18T04:00:25.6870167Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpipcldngq
2022-05-18T04:00:25.6871396Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpipcldngq/_remote_module_non_scriptable.py
2022-05-18T04:00:25.7148612Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprhk9ectc
2022-05-18T04:00:25.7150039Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprhk9ectc/_remote_module_non_scriptable.py
2022-05-18T04:00:25.7277174Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpojzhaq1b
2022-05-18T04:00:25.7278577Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpojzhaq1b/_remote_module_non_scriptable.py
2022-05-18T04:00:25.9200161Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:00:25.9394290Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:00:25.9619362Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:00:25.9759998Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:00:26.2135041Z ok (1.526s)
2022-05-18T04:00:26.2135457Z 
2022-05-18T04:00:26.2135808Z ----------------------------------------------------------------------
2022-05-18T04:00:26.2136302Z Ran 1 test in 1.526s
2022-05-18T04:00:26.2136511Z 
2022-05-18T04:00:26.2136613Z OK
2022-05-18T04:00:26.2136737Z 
2022-05-18T04:00:26.2136835Z Generating XML reports...
2022-05-18T04:00:26.2171228Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040024.xml
2022-05-18T04:00:26.9637670Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb5f3khe8
2022-05-18T04:00:26.9638434Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb5f3khe8/_remote_module_non_scriptable.py
2022-05-18T04:00:27.2164098Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:00:27.2173203Z 
2022-05-18T04:00:27.2173294Z Running tests...
2022-05-18T04:00:27.2174262Z ----------------------------------------------------------------------
2022-05-18T04:00:27.5346633Z   test_builtin_remote_ret (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13303
2022-05-18T04:00:27.5369474Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13304
2022-05-18T04:00:27.5393075Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13305
2022-05-18T04:00:27.5417705Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 13306
2022-05-18T04:00:28.1261611Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprrr99lrx
2022-05-18T04:00:28.1262754Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprrr99lrx/_remote_module_non_scriptable.py
2022-05-18T04:00:28.1652477Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx8soo_6d
2022-05-18T04:00:28.1653291Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph076t_y6
2022-05-18T04:00:28.1654005Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx8soo_6d/_remote_module_non_scriptable.py
2022-05-18T04:00:28.1654757Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph076t_y6/_remote_module_non_scriptable.py
2022-05-18T04:00:28.1728513Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwh9u7ls4
2022-05-18T04:00:28.1729457Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwh9u7ls4/_remote_module_non_scriptable.py
2022-05-18T04:00:28.3758603Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:00:28.4144449Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:00:28.4153916Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:00:28.4185145Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:00:28.9457908Z ok (1.728s)
2022-05-18T04:00:28.9458083Z 
2022-05-18T04:00:28.9458406Z ----------------------------------------------------------------------
2022-05-18T04:00:28.9458700Z Ran 1 test in 1.728s
2022-05-18T04:00:28.9458818Z 
2022-05-18T04:00:28.9458882Z OK
2022-05-18T04:00:28.9458975Z 
2022-05-18T04:00:28.9459058Z Generating XML reports...
2022-05-18T04:00:28.9492673Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040027.xml
2022-05-18T04:00:29.7264024Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv_ntqlze
2022-05-18T04:00:29.7264488Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv_ntqlze/_remote_module_non_scriptable.py
2022-05-18T04:00:29.9790623Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:00:29.9800241Z 
2022-05-18T04:00:29.9800318Z Running tests...
2022-05-18T04:00:29.9801390Z ----------------------------------------------------------------------
2022-05-18T04:00:30.2928808Z   test_builtin_remote_self (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13522
2022-05-18T04:00:30.2951109Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13523
2022-05-18T04:00:30.2974717Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13524
2022-05-18T04:00:30.2998478Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 13525
2022-05-18T04:00:30.9668418Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3wmnixin
2022-05-18T04:00:30.9669157Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3wmnixin/_remote_module_non_scriptable.py
2022-05-18T04:00:31.0239920Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplw64ajnv
2022-05-18T04:00:31.0240729Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplw64ajnv/_remote_module_non_scriptable.py
2022-05-18T04:00:31.0317106Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptgtzm94o
2022-05-18T04:00:31.0318678Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptgtzm94o/_remote_module_non_scriptable.py
2022-05-18T04:00:31.0320599Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnbfk3uwm
2022-05-18T04:00:31.0322921Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnbfk3uwm/_remote_module_non_scriptable.py
2022-05-18T04:00:31.2164763Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:00:31.2726551Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:00:31.2771357Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:00:31.2796317Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:00:31.7038743Z ok (1.724s)
2022-05-18T04:00:31.7038964Z 
2022-05-18T04:00:31.7039493Z ----------------------------------------------------------------------
2022-05-18T04:00:31.7039924Z Ran 1 test in 1.724s
2022-05-18T04:00:31.7040026Z 
2022-05-18T04:00:31.7040088Z OK
2022-05-18T04:00:31.7040180Z 
2022-05-18T04:00:31.7040273Z Generating XML reports...
2022-05-18T04:00:31.7073632Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040029.xml
2022-05-18T04:00:32.4842505Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppn8l9yk8
2022-05-18T04:00:32.4843265Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppn8l9yk8/_remote_module_non_scriptable.py
2022-05-18T04:00:32.7380332Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:00:32.7389359Z 
2022-05-18T04:00:32.7389461Z Running tests...
2022-05-18T04:00:32.7390268Z ----------------------------------------------------------------------
2022-05-18T04:00:32.7400125Z   test_call_method_on_rref (__main__.TensorPipeRpcTest)
2022-05-18T04:00:33.0568562Z Tests that it is possible to call an instance method on a remote objet ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13741
2022-05-18T04:00:33.0591700Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13742
2022-05-18T04:00:33.0615041Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13743
2022-05-18T04:00:33.0639267Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 13744
2022-05-18T04:00:33.7140789Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcc382ui0
2022-05-18T04:00:33.7141634Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcc382ui0/_remote_module_non_scriptable.py
2022-05-18T04:00:33.7576036Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1s2owzp4
2022-05-18T04:00:33.7576799Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1s2owzp4/_remote_module_non_scriptable.py
2022-05-18T04:00:33.7929978Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpavlbhkab
2022-05-18T04:00:33.7930830Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpavlbhkab/_remote_module_non_scriptable.py
2022-05-18T04:00:33.7936363Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjttmn19y
2022-05-18T04:00:33.7938640Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjttmn19y/_remote_module_non_scriptable.py
2022-05-18T04:00:33.9639936Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:00:34.0050546Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:00:34.0414115Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:00:34.0418496Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:00:34.5686348Z ok (1.829s)
2022-05-18T04:00:34.5686591Z 
2022-05-18T04:00:34.5687416Z ----------------------------------------------------------------------
2022-05-18T04:00:34.5687745Z Ran 1 test in 1.830s
2022-05-18T04:00:34.5687864Z 
2022-05-18T04:00:34.5687926Z OK
2022-05-18T04:00:34.5688019Z 
2022-05-18T04:00:34.5688180Z Generating XML reports...
2022-05-18T04:00:34.5722185Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040032.xml
2022-05-18T04:00:35.3490257Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp65kkvspk
2022-05-18T04:00:35.3491207Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp65kkvspk/_remote_module_non_scriptable.py
2022-05-18T04:00:35.6033762Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:00:35.6043232Z 
2022-05-18T04:00:35.6043375Z Running tests...
2022-05-18T04:00:35.6043815Z ----------------------------------------------------------------------
2022-05-18T04:00:35.9180882Z   test_callback_chain (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13960
2022-05-18T04:00:35.9205057Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13961
2022-05-18T04:00:35.9227823Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13962
2022-05-18T04:00:35.9251999Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 13963
2022-05-18T04:00:36.5399340Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpean5_ons
2022-05-18T04:00:36.5400486Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpean5_ons/_remote_module_non_scriptable.py
2022-05-18T04:00:36.5631054Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqsv4hz7n
2022-05-18T04:00:36.5632085Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqsv4hz7n/_remote_module_non_scriptable.py
2022-05-18T04:00:36.5714732Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7pkad1or
2022-05-18T04:00:36.5715890Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7pkad1or/_remote_module_non_scriptable.py
2022-05-18T04:00:36.5740740Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjutds2xq
2022-05-18T04:00:36.5742033Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjutds2xq/_remote_module_non_scriptable.py
2022-05-18T04:00:36.7894489Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:00:36.8136529Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:00:36.8233659Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:00:36.8241091Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:00:37.3292283Z ok (1.725s)
2022-05-18T04:00:37.3292536Z 
2022-05-18T04:00:37.3293118Z ----------------------------------------------------------------------
2022-05-18T04:00:37.3293546Z Ran 1 test in 1.725s
2022-05-18T04:00:37.3293756Z 
2022-05-18T04:00:37.3293866Z OK
2022-05-18T04:00:37.3294034Z 
2022-05-18T04:00:37.3294211Z Generating XML reports...
2022-05-18T04:00:37.3328506Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040035.xml
2022-05-18T04:00:38.1071581Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphmpk2_kw
2022-05-18T04:00:38.1072129Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphmpk2_kw/_remote_module_non_scriptable.py
2022-05-18T04:00:38.3609681Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:00:38.3619036Z 
2022-05-18T04:00:38.3619162Z Running tests...
2022-05-18T04:00:38.3619764Z ----------------------------------------------------------------------
2022-05-18T04:00:38.6831513Z   test_callback_in_rpc (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14179
2022-05-18T04:00:38.6856077Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14180
2022-05-18T04:00:38.6879170Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14181
2022-05-18T04:00:38.6903755Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 14182
2022-05-18T04:00:39.2963068Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmimkhvpi
2022-05-18T04:00:39.2963818Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmimkhvpi/_remote_module_non_scriptable.py
2022-05-18T04:00:39.3198544Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdx6bax_e
2022-05-18T04:00:39.3199506Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdx6bax_e/_remote_module_non_scriptable.py
2022-05-18T04:00:39.3519856Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplxf_uinj
2022-05-18T04:00:39.3520634Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplxf_uinj/_remote_module_non_scriptable.py
2022-05-18T04:00:39.3550544Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0d13uu1g
2022-05-18T04:00:39.3551462Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0d13uu1g/_remote_module_non_scriptable.py
2022-05-18T04:00:39.5443837Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:00:39.5675226Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:00:39.6023485Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:00:39.6036655Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:00:39.9942247Z ok (1.632s)
2022-05-18T04:00:39.9942465Z 
2022-05-18T04:00:39.9942794Z ----------------------------------------------------------------------
2022-05-18T04:00:39.9943276Z Ran 1 test in 1.632s
2022-05-18T04:00:39.9943393Z 
2022-05-18T04:00:39.9943455Z OK
2022-05-18T04:00:39.9943534Z 
2022-05-18T04:00:39.9943645Z Generating XML reports...
2022-05-18T04:00:39.9976831Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040038.xml
2022-05-18T04:00:40.7742857Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpviis01m2
2022-05-18T04:00:40.7743812Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpviis01m2/_remote_module_non_scriptable.py
2022-05-18T04:00:41.0272352Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:00:41.0282027Z 
2022-05-18T04:00:41.0282127Z Running tests...
2022-05-18T04:00:41.0282581Z ----------------------------------------------------------------------
2022-05-18T04:00:41.3476840Z   test_callback_multi (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14398
2022-05-18T04:00:41.3499997Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14399
2022-05-18T04:00:41.3522765Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14400
2022-05-18T04:00:41.3547086Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 14401
2022-05-18T04:00:41.9704405Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn52ea320
2022-05-18T04:00:41.9705145Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn52ea320/_remote_module_non_scriptable.py
2022-05-18T04:00:41.9890039Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp09_h43rj
2022-05-18T04:00:41.9890952Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp09_h43rj/_remote_module_non_scriptable.py
2022-05-18T04:00:41.9891621Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa3vt0ttq
2022-05-18T04:00:41.9893691Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa3vt0ttq/_remote_module_non_scriptable.py
2022-05-18T04:00:41.9897192Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu1rm12j5
2022-05-18T04:00:41.9899759Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu1rm12j5/_remote_module_non_scriptable.py
2022-05-18T04:00:42.2183784Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:00:42.2359824Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:00:42.2370775Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:00:42.2383915Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:00:42.7587894Z ok (1.730s)
2022-05-18T04:00:42.7588144Z 
2022-05-18T04:00:42.7588661Z ----------------------------------------------------------------------
2022-05-18T04:00:42.7589083Z Ran 1 test in 1.731s
2022-05-18T04:00:42.7589201Z 
2022-05-18T04:00:42.7589266Z OK
2022-05-18T04:00:42.7589359Z 
2022-05-18T04:00:42.7589458Z Generating XML reports...
2022-05-18T04:00:42.7624313Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040041.xml
2022-05-18T04:00:43.5427957Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr_xy4gy6
2022-05-18T04:00:43.5428852Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr_xy4gy6/_remote_module_non_scriptable.py
2022-05-18T04:00:43.7966109Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:00:43.7976107Z 
2022-05-18T04:00:43.7976208Z Running tests...
2022-05-18T04:00:43.7976761Z ----------------------------------------------------------------------
2022-05-18T04:00:44.1123319Z   test_callback_none (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14617
2022-05-18T04:00:44.1146136Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14618
2022-05-18T04:00:44.1168948Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14619
2022-05-18T04:00:44.1192838Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 14620
2022-05-18T04:00:44.7341062Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpchan351q
2022-05-18T04:00:44.7341866Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpchan351q/_remote_module_non_scriptable.py
2022-05-18T04:00:44.7463208Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcp_in3zm
2022-05-18T04:00:44.7465723Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjiq5tu6j
2022-05-18T04:00:44.7466489Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcp_in3zm/_remote_module_non_scriptable.py
2022-05-18T04:00:44.7467188Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjiq5tu6j/_remote_module_non_scriptable.py
2022-05-18T04:00:44.7680196Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp75jg60tz
2022-05-18T04:00:44.7681099Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp75jg60tz/_remote_module_non_scriptable.py
2022-05-18T04:00:44.9847609Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:00:44.9941358Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:00:44.9943066Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:00:45.0185888Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:00:45.2291415Z On WorkerInfo(id=1, name=worker1):
2022-05-18T04:00:45.2292186Z ValueError('Expected error')
2022-05-18T04:00:45.2293122Z Traceback (most recent call last):
2022-05-18T04:00:45.2294028Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function
2022-05-18T04:00:45.2294892Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-05-18T04:00:45.2295864Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 396, in raise_func
2022-05-18T04:00:45.2296540Z     raise ValueError(expected_err)
2022-05-18T04:00:45.2296993Z ValueError: Expected error
2022-05-18T04:00:45.2297271Z 
2022-05-18T04:00:45.2489539Z On WorkerInfo(id=0, name=worker0):
2022-05-18T04:00:45.2490066Z ValueError('Expected error')
2022-05-18T04:00:45.2490552Z Traceback (most recent call last):
2022-05-18T04:00:45.2491246Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function
2022-05-18T04:00:45.2494409Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-05-18T04:00:45.2495604Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 396, in raise_func
2022-05-18T04:00:45.2496161Z     raise ValueError(expected_err)
2022-05-18T04:00:45.2496673Z ValueError: Expected error
2022-05-18T04:00:45.2496905Z 
2022-05-18T04:00:45.2605882Z On WorkerInfo(id=3, name=worker3):
2022-05-18T04:00:45.2606494Z ValueError('Expected error')
2022-05-18T04:00:45.2606778Z Traceback (most recent call last):
2022-05-18T04:00:45.2607206Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function
2022-05-18T04:00:45.2607550Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-05-18T04:00:45.2607999Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 396, in raise_func
2022-05-18T04:00:45.2608292Z     raise ValueError(expected_err)
2022-05-18T04:00:45.2608502Z ValueError: Expected error
2022-05-18T04:00:45.2608640Z 
2022-05-18T04:00:45.2646082Z On WorkerInfo(id=2, name=worker2):
2022-05-18T04:00:45.2646553Z ValueError('Expected error')
2022-05-18T04:00:45.2646758Z Traceback (most recent call last):
2022-05-18T04:00:45.2647174Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function
2022-05-18T04:00:45.2647561Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-05-18T04:00:45.2648039Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 396, in raise_func
2022-05-18T04:00:45.2648354Z     raise ValueError(expected_err)
2022-05-18T04:00:45.2648564Z ValueError: Expected error
2022-05-18T04:00:45.2648696Z 
2022-05-18T04:00:45.5234024Z ok (1.725s)
2022-05-18T04:00:45.5234241Z 
2022-05-18T04:00:45.5234786Z ----------------------------------------------------------------------
2022-05-18T04:00:45.5235158Z Ran 1 test in 1.726s
2022-05-18T04:00:45.5235263Z 
2022-05-18T04:00:45.5235326Z OK
2022-05-18T04:00:45.5235426Z 
2022-05-18T04:00:45.5235517Z Generating XML reports...
2022-05-18T04:00:45.5269211Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040043.xml
2022-05-18T04:00:46.2993002Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjc21b431
2022-05-18T04:00:46.2993877Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjc21b431/_remote_module_non_scriptable.py
2022-05-18T04:00:46.5524245Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:00:46.5533796Z 
2022-05-18T04:00:46.5533935Z Running tests...
2022-05-18T04:00:46.5534517Z ----------------------------------------------------------------------
2022-05-18T04:00:46.8768445Z   test_callback_simple (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14836
2022-05-18T04:00:46.8790866Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14837
2022-05-18T04:00:46.8813749Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14838
2022-05-18T04:00:46.8839304Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 14839
2022-05-18T04:00:47.4771106Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfwwe7840
2022-05-18T04:00:47.4772447Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfwwe7840/_remote_module_non_scriptable.py
2022-05-18T04:00:47.5254197Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx_4n8ipn
2022-05-18T04:00:47.5254988Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx_4n8ipn/_remote_module_non_scriptable.py
2022-05-18T04:00:47.5567047Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqxj4ryxt
2022-05-18T04:00:47.5568096Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqxj4ryxt/_remote_module_non_scriptable.py
2022-05-18T04:00:47.5706573Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkmernk2t
2022-05-18T04:00:47.5707854Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkmernk2t/_remote_module_non_scriptable.py
2022-05-18T04:00:47.7261865Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:00:47.7727932Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:00:47.8061258Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:00:47.8181353Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:00:48.2880418Z ok (1.734s)
2022-05-18T04:00:48.2880696Z 
2022-05-18T04:00:48.2881187Z ----------------------------------------------------------------------
2022-05-18T04:00:48.2881458Z Ran 1 test in 1.735s
2022-05-18T04:00:48.2881571Z 
2022-05-18T04:00:48.2881619Z OK
2022-05-18T04:00:48.2881737Z 
2022-05-18T04:00:48.2881829Z Generating XML reports...
2022-05-18T04:00:48.2915957Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040046.xml
2022-05-18T04:00:49.1439145Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvjg7to3h
2022-05-18T04:00:49.1439788Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvjg7to3h/_remote_module_non_scriptable.py
2022-05-18T04:00:49.3982138Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:00:49.3991918Z 
2022-05-18T04:00:49.3992051Z Running tests...
2022-05-18T04:00:49.3992510Z ----------------------------------------------------------------------
2022-05-18T04:00:49.7181006Z   test_callback_with_error (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15055
2022-05-18T04:00:49.7205076Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15056
2022-05-18T04:00:49.7228961Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15057
2022-05-18T04:00:49.7255002Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 15058
2022-05-18T04:00:50.3125887Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppmv5b_xx
2022-05-18T04:00:50.3127251Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppmv5b_xx/_remote_module_non_scriptable.py
2022-05-18T04:00:50.3527367Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprp4r5qu7
2022-05-18T04:00:50.3528102Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprp4r5qu7/_remote_module_non_scriptable.py
2022-05-18T04:00:50.3749273Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm8dgnsc4
2022-05-18T04:00:50.3750021Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm8dgnsc4/_remote_module_non_scriptable.py
2022-05-18T04:00:50.3801999Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnvy39c60
2022-05-18T04:00:50.3803842Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnvy39c60/_remote_module_non_scriptable.py
2022-05-18T04:00:50.5665692Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:00:50.6052777Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:00:50.6302725Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:00:50.6334004Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:00:50.8493105Z On WorkerInfo(id=1, name=worker1):
2022-05-18T04:00:50.8493682Z ValueError('Expected error')
2022-05-18T04:00:50.8493999Z Traceback (most recent call last):
2022-05-18T04:00:50.8494680Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function
2022-05-18T04:00:50.8496619Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-05-18T04:00:50.8497804Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 396, in raise_func
2022-05-18T04:00:50.8498494Z     raise ValueError(expected_err)
2022-05-18T04:00:50.8498928Z ValueError: Expected error
2022-05-18T04:00:50.8499199Z 
2022-05-18T04:00:50.8648025Z On WorkerInfo(id=2, name=worker2):
2022-05-18T04:00:50.8648546Z ValueError('Expected error')
2022-05-18T04:00:50.8648855Z Traceback (most recent call last):
2022-05-18T04:00:50.8651002Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function
2022-05-18T04:00:50.8651768Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-05-18T04:00:50.8652756Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 396, in raise_func
2022-05-18T04:00:50.8653478Z     raise ValueError(expected_err)
2022-05-18T04:00:50.8653972Z ValueError: Expected error
2022-05-18T04:00:50.8654246Z 
2022-05-18T04:00:50.8690435Z On WorkerInfo(id=0, name=worker0):
2022-05-18T04:00:50.8691218Z ValueError('Expected error')
2022-05-18T04:00:50.8691656Z Traceback (most recent call last):
2022-05-18T04:00:50.8692529Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function
2022-05-18T04:00:50.8693313Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-05-18T04:00:50.8694271Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 396, in raise_func
2022-05-18T04:00:50.8694944Z     raise ValueError(expected_err)
2022-05-18T04:00:50.8695396Z ValueError: Expected error
2022-05-18T04:00:50.8695659Z 
2022-05-18T04:00:50.8776583Z On WorkerInfo(id=3, name=worker3):
2022-05-18T04:00:50.8777263Z ValueError('Expected error')
2022-05-18T04:00:50.8777717Z Traceback (most recent call last):
2022-05-18T04:00:50.8778615Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function
2022-05-18T04:00:50.8779328Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-05-18T04:00:50.8780307Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 396, in raise_func
2022-05-18T04:00:50.8780966Z     raise ValueError(expected_err)
2022-05-18T04:00:50.8781413Z ValueError: Expected error
2022-05-18T04:00:50.8781655Z 
2022-05-18T04:00:51.1294756Z ok (1.730s)
2022-05-18T04:00:51.1295052Z 
2022-05-18T04:00:51.1295526Z ----------------------------------------------------------------------
2022-05-18T04:00:51.1295776Z Ran 1 test in 1.730s
2022-05-18T04:00:51.1295894Z 
2022-05-18T04:00:51.1295943Z OK
2022-05-18T04:00:51.1296035Z 
2022-05-18T04:00:51.1296127Z Generating XML reports...
2022-05-18T04:00:51.1333764Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040049.xml
2022-05-18T04:00:51.9953155Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyhzdgrfc
2022-05-18T04:00:51.9954057Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyhzdgrfc/_remote_module_non_scriptable.py
2022-05-18T04:00:52.2620560Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:00:52.2630182Z 
2022-05-18T04:00:52.2630330Z Running tests...
2022-05-18T04:00:52.2630806Z ----------------------------------------------------------------------
2022-05-18T04:00:52.6092959Z   test_callback_with_ret (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15274
2022-05-18T04:00:52.6117652Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15275
2022-05-18T04:00:52.6143142Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15276
2022-05-18T04:00:52.6168020Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 15277
2022-05-18T04:00:53.2719146Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu7lagypx
2022-05-18T04:00:53.2719968Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu7lagypx/_remote_module_non_scriptable.py
2022-05-18T04:00:53.2835411Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1ta6oh2v
2022-05-18T04:00:53.2836143Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1ta6oh2v/_remote_module_non_scriptable.py
2022-05-18T04:00:53.2966194Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8whpd7fm
2022-05-18T04:00:53.2966866Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8whpd7fm/_remote_module_non_scriptable.py
2022-05-18T04:00:53.3145131Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc70o9dx6
2022-05-18T04:00:53.3145865Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc70o9dx6/_remote_module_non_scriptable.py
2022-05-18T04:00:53.5276486Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:00:53.5413973Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:00:53.5528651Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:00:53.5699772Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:00:54.0209038Z ok (1.757s)
2022-05-18T04:00:54.0209197Z 
2022-05-18T04:00:54.0209524Z ----------------------------------------------------------------------
2022-05-18T04:00:54.0209776Z Ran 1 test in 1.758s
2022-05-18T04:00:54.0209892Z 
2022-05-18T04:00:54.0209940Z OK
2022-05-18T04:00:54.0210032Z 
2022-05-18T04:00:54.0210125Z Generating XML reports...
2022-05-18T04:00:54.0243456Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040052.xml
2022-05-18T04:00:54.8205193Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpva1r6lru
2022-05-18T04:00:54.8205885Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpva1r6lru/_remote_module_non_scriptable.py
2022-05-18T04:00:55.0811201Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:00:55.0820957Z 
2022-05-18T04:00:55.0821132Z Running tests...
2022-05-18T04:00:55.0821470Z ----------------------------------------------------------------------
2022-05-18T04:00:55.4198579Z   test_callback_wrong_arg_num (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15493
2022-05-18T04:00:55.4223524Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15494
2022-05-18T04:00:55.4247679Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15495
2022-05-18T04:00:55.4272994Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 15496
2022-05-18T04:00:56.1118573Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptlrm2d2j
2022-05-18T04:00:56.1119555Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptlrm2d2j/_remote_module_non_scriptable.py
2022-05-18T04:00:56.1157255Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5z18h4xc
2022-05-18T04:00:56.1158564Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5z18h4xc/_remote_module_non_scriptable.py
2022-05-18T04:00:56.1871576Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppeyy05w9
2022-05-18T04:00:56.1872318Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfmcaz6gz
2022-05-18T04:00:56.1872998Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppeyy05w9/_remote_module_non_scriptable.py
2022-05-18T04:00:56.1873727Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfmcaz6gz/_remote_module_non_scriptable.py
2022-05-18T04:00:56.3692779Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:00:56.3825185Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:00:56.4419154Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:00:56.4494611Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:00:56.9315037Z ok (1.849s)
2022-05-18T04:00:56.9315328Z 
2022-05-18T04:00:56.9315809Z ----------------------------------------------------------------------
2022-05-18T04:00:56.9316066Z Ran 1 test in 1.849s
2022-05-18T04:00:56.9316181Z 
2022-05-18T04:00:56.9316245Z OK
2022-05-18T04:00:56.9316338Z 
2022-05-18T04:00:56.9316434Z Generating XML reports...
2022-05-18T04:00:56.9349511Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040055.xml
2022-05-18T04:00:57.7442899Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx9yissh_
2022-05-18T04:00:57.7455305Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx9yissh_/_remote_module_non_scriptable.py
2022-05-18T04:00:58.0000196Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:00:58.0009608Z 
2022-05-18T04:00:58.0009726Z Running tests...
2022-05-18T04:00:58.0010797Z ----------------------------------------------------------------------
2022-05-18T04:00:58.3315610Z   test_callback_wrong_arg_type (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15712
2022-05-18T04:00:58.3340166Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15713
2022-05-18T04:00:58.3364842Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15714
2022-05-18T04:00:58.3390237Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 15715
2022-05-18T04:00:59.0247344Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbu3ttafm
2022-05-18T04:00:59.0255781Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbu3ttafm/_remote_module_non_scriptable.py
2022-05-18T04:00:59.0584523Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1lmhwq93
2022-05-18T04:00:59.0585574Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1lmhwq93/_remote_module_non_scriptable.py
2022-05-18T04:00:59.0962146Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2mparsw6
2022-05-18T04:00:59.0963990Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2mparsw6/_remote_module_non_scriptable.py
2022-05-18T04:00:59.1260872Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplkifk5el
2022-05-18T04:00:59.1261876Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplkifk5el/_remote_module_non_scriptable.py
2022-05-18T04:00:59.2876709Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:00:59.3602653Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:00:59.3798193Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:00:59.3884166Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:00:59.8432131Z ok (1.842s)
2022-05-18T04:00:59.8432638Z 
2022-05-18T04:00:59.8432955Z ----------------------------------------------------------------------
2022-05-18T04:00:59.8442025Z Ran 1 test in 1.842s
2022-05-18T04:00:59.8442154Z 
2022-05-18T04:00:59.8442217Z OK
2022-05-18T04:00:59.8442311Z 
2022-05-18T04:00:59.8442395Z Generating XML reports...
2022-05-18T04:00:59.8467470Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040057.xml
2022-05-18T04:01:00.7053832Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3fxv5mca
2022-05-18T04:01:00.7054778Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3fxv5mca/_remote_module_non_scriptable.py
2022-05-18T04:01:00.9728268Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:01:00.9737671Z 
2022-05-18T04:01:00.9737807Z Running tests...
2022-05-18T04:01:00.9738422Z ----------------------------------------------------------------------
2022-05-18T04:01:01.3173469Z   test_cannot_infer_backend_from_options (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15931
2022-05-18T04:01:01.3198244Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15932
2022-05-18T04:01:01.3223250Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15933
2022-05-18T04:01:01.3249403Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 15934
2022-05-18T04:01:01.9717360Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg6tntk73
2022-05-18T04:01:01.9718215Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg6tntk73/_remote_module_non_scriptable.py
2022-05-18T04:01:02.0547265Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp904dije2
2022-05-18T04:01:02.0548211Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp904dije2/_remote_module_non_scriptable.py
2022-05-18T04:01:02.0802529Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpowyrt0b7
2022-05-18T04:01:02.0803239Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpowyrt0b7/_remote_module_non_scriptable.py
2022-05-18T04:01:02.1030358Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8gz8fwp5
2022-05-18T04:01:02.1031212Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8gz8fwp5/_remote_module_non_scriptable.py
2022-05-18T04:01:02.2390180Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:01:02.3176327Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:01:02.3441319Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:01:02.3697989Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:01:02.5285956Z ok (1.555s)
2022-05-18T04:01:02.5286184Z 
2022-05-18T04:01:02.5286711Z ----------------------------------------------------------------------
2022-05-18T04:01:02.5287079Z Ran 1 test in 1.555s
2022-05-18T04:01:02.5287195Z 
2022-05-18T04:01:02.5287256Z OK
2022-05-18T04:01:02.5287348Z 
2022-05-18T04:01:02.5287445Z Generating XML reports...
2022-05-18T04:01:02.5321889Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040100.xml
2022-05-18T04:01:03.3923646Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgt606fuj
2022-05-18T04:01:03.3924721Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgt606fuj/_remote_module_non_scriptable.py
2022-05-18T04:01:03.6657760Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:01:03.6667033Z 
2022-05-18T04:01:03.6667127Z Running tests...
2022-05-18T04:01:03.6667598Z ----------------------------------------------------------------------
2022-05-18T04:01:04.0149099Z   test_deadlock (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15986
2022-05-18T04:01:04.0173023Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15987
2022-05-18T04:01:04.0197959Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15988
2022-05-18T04:01:04.0223239Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 15989
2022-05-18T04:01:04.6999972Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzv6pgba1
2022-05-18T04:01:04.7001052Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzv6pgba1/_remote_module_non_scriptable.py
2022-05-18T04:01:04.7146981Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg8bhgpt4
2022-05-18T04:01:04.7148258Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg8bhgpt4/_remote_module_non_scriptable.py
2022-05-18T04:01:04.7278472Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9bc37kb5
2022-05-18T04:01:04.7279209Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9bc37kb5/_remote_module_non_scriptable.py
2022-05-18T04:01:04.7501941Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpncmnu3wl
2022-05-18T04:01:04.7503311Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpncmnu3wl/_remote_module_non_scriptable.py
2022-05-18T04:01:04.9628250Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:01:04.9792349Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:01:04.9934764Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:01:05.0138491Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:01:06.2722260Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:01:06.2824598Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T04:01:06.2825462Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:01:06.2826169Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:01:06.2827302Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:01:06.2828992Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:01:06.2830120Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:01:06.2834390Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:01:06.5283546Z ok (2.861s)
2022-05-18T04:01:06.5283790Z 
2022-05-18T04:01:06.5284269Z ----------------------------------------------------------------------
2022-05-18T04:01:06.5284647Z Ran 1 test in 2.861s
2022-05-18T04:01:06.5284837Z 
2022-05-18T04:01:06.5284915Z OK
2022-05-18T04:01:06.5285068Z 
2022-05-18T04:01:06.5285213Z Generating XML reports...
2022-05-18T04:01:06.5324072Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040103.xml
2022-05-18T04:01:07.3983310Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl8vfuw0j
2022-05-18T04:01:07.3984112Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl8vfuw0j/_remote_module_non_scriptable.py
2022-05-18T04:01:07.6701595Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:01:07.6712562Z 
2022-05-18T04:01:07.6712884Z Running tests...
2022-05-18T04:01:07.6713473Z ----------------------------------------------------------------------
2022-05-18T04:01:08.0182767Z   test_debug_info (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16217
2022-05-18T04:01:08.0206370Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16218
2022-05-18T04:01:08.0231007Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16219
2022-05-18T04:01:08.0256856Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 16220
2022-05-18T04:01:08.7119225Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyhae7yin
2022-05-18T04:01:08.7120415Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyhae7yin/_remote_module_non_scriptable.py
2022-05-18T04:01:08.7600138Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp106d9ate
2022-05-18T04:01:08.7600898Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp106d9ate/_remote_module_non_scriptable.py
2022-05-18T04:01:08.7654933Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv1vgbra1
2022-05-18T04:01:08.7655751Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv1vgbra1/_remote_module_non_scriptable.py
2022-05-18T04:01:08.8240791Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpasj_7rac
2022-05-18T04:01:08.8241555Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpasj_7rac/_remote_module_non_scriptable.py
2022-05-18T04:01:08.9751650Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:01:09.0238861Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:01:09.0256517Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:01:09.0870062Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:01:09.5296912Z ok (1.858s)
2022-05-18T04:01:09.5297166Z 
2022-05-18T04:01:09.5297672Z ----------------------------------------------------------------------
2022-05-18T04:01:09.5298137Z Ran 1 test in 1.858s
2022-05-18T04:01:09.5298332Z 
2022-05-18T04:01:09.5298443Z OK
2022-05-18T04:01:09.5298610Z 
2022-05-18T04:01:09.5298777Z Generating XML reports...
2022-05-18T04:01:09.5332373Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040107.xml
2022-05-18T04:01:10.3739524Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu8w0nm9c
2022-05-18T04:01:10.3740416Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu8w0nm9c/_remote_module_non_scriptable.py
2022-05-18T04:01:10.6400864Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:01:10.6410440Z 
2022-05-18T04:01:10.6410530Z Running tests...
2022-05-18T04:01:10.6411015Z ----------------------------------------------------------------------
2022-05-18T04:01:10.6421875Z   test_default_timeout_used (__main__.TensorPipeRpcTest)
2022-05-18T04:01:10.9831002Z Tests that if no timeout is passed into rpc_async and rpc_sync, then the ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16440
2022-05-18T04:01:10.9854995Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16441
2022-05-18T04:01:10.9878721Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16442
2022-05-18T04:01:10.9903718Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 16443
2022-05-18T04:01:11.7015165Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0vj0uj29
2022-05-18T04:01:11.7016343Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0vj0uj29/_remote_module_non_scriptable.py
2022-05-18T04:01:11.7521461Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl9mxk52m
2022-05-18T04:01:11.7522145Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpiq0fwqv1
2022-05-18T04:01:11.7522838Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl9mxk52m/_remote_module_non_scriptable.py
2022-05-18T04:01:11.7523491Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpiq0fwqv1/_remote_module_non_scriptable.py
2022-05-18T04:01:11.7677109Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpan4afn14
2022-05-18T04:01:11.7677900Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpan4afn14/_remote_module_non_scriptable.py
2022-05-18T04:01:11.9597165Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:01:12.0084652Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:01:12.0090868Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:01:12.0254804Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:01:14.4977061Z ok (3.856s)
2022-05-18T04:01:14.4977291Z 
2022-05-18T04:01:14.4977738Z ----------------------------------------------------------------------
2022-05-18T04:01:14.4978127Z Ran 1 test in 3.857s
2022-05-18T04:01:14.4978313Z 
2022-05-18T04:01:14.4978393Z OK
2022-05-18T04:01:14.4978551Z 
2022-05-18T04:01:14.4978692Z Generating XML reports...
2022-05-18T04:01:14.5012451Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040110.xml
2022-05-18T04:01:15.2870237Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl3ymxpv8
2022-05-18T04:01:15.2870748Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl3ymxpv8/_remote_module_non_scriptable.py
2022-05-18T04:01:15.5419985Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:01:15.5430165Z 
2022-05-18T04:01:15.5430310Z Running tests...
2022-05-18T04:01:15.5430712Z ----------------------------------------------------------------------
2022-05-18T04:01:15.8662169Z   test_disable_gil_profiling (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16659
2022-05-18T04:01:15.8685790Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16660
2022-05-18T04:01:15.8709152Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16661
2022-05-18T04:01:15.8733044Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 16662
2022-05-18T04:01:16.5254784Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5rpi0vfm
2022-05-18T04:01:16.5255543Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5rpi0vfm/_remote_module_non_scriptable.py
2022-05-18T04:01:16.5399504Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3tiel28c
2022-05-18T04:01:16.5400572Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3tiel28c/_remote_module_non_scriptable.py
2022-05-18T04:01:16.5503299Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx_n9xrgu
2022-05-18T04:01:16.5504830Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx_n9xrgu/_remote_module_non_scriptable.py
2022-05-18T04:01:16.6056271Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeje0e7mt
2022-05-18T04:01:16.6057313Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeje0e7mt/_remote_module_non_scriptable.py
2022-05-18T04:01:16.7805455Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:01:16.7904497Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:01:16.8016388Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:01:16.8551249Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:01:17.2773271Z ok (1.734s)
2022-05-18T04:01:17.2773433Z 
2022-05-18T04:01:17.2773779Z ----------------------------------------------------------------------
2022-05-18T04:01:17.2774037Z Ran 1 test in 1.734s
2022-05-18T04:01:17.2774156Z 
2022-05-18T04:01:17.2775192Z OK
2022-05-18T04:01:17.2775354Z 
2022-05-18T04:01:17.2775462Z Generating XML reports...
2022-05-18T04:01:17.2809714Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040115.xml
2022-05-18T04:01:18.0626960Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1verr6yp
2022-05-18T04:01:18.0627791Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1verr6yp/_remote_module_non_scriptable.py
2022-05-18T04:01:18.3164194Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:01:18.3174127Z 
2022-05-18T04:01:18.3174268Z Running tests...
2022-05-18T04:01:18.3174773Z ----------------------------------------------------------------------
2022-05-18T04:01:18.6337083Z   test_dist_init_decorator (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16878
2022-05-18T04:01:18.6360133Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16879
2022-05-18T04:01:18.6383119Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16880
2022-05-18T04:01:18.6406759Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 16881
2022-05-18T04:01:19.2115391Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8vex1hqd
2022-05-18T04:01:19.2116169Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8vex1hqd/_remote_module_non_scriptable.py
2022-05-18T04:01:19.2248101Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6vprn9zp
2022-05-18T04:01:19.2248845Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6vprn9zp/_remote_module_non_scriptable.py
2022-05-18T04:01:19.2403119Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp280vnwdw
2022-05-18T04:01:19.2403918Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp280vnwdw/_remote_module_non_scriptable.py
2022-05-18T04:01:19.2970306Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeuxeovm0
2022-05-18T04:01:19.2971143Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeuxeovm0/_remote_module_non_scriptable.py
2022-05-18T04:01:19.4657573Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:01:19.4770378Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:01:19.4909967Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:01:19.5497909Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:01:19.9445041Z ok (1.627s)
2022-05-18T04:01:19.9445274Z 
2022-05-18T04:01:19.9445828Z ----------------------------------------------------------------------
2022-05-18T04:01:19.9446188Z Ran 1 test in 1.627s
2022-05-18T04:01:19.9446304Z 
2022-05-18T04:01:19.9446352Z OK
2022-05-18T04:01:19.9446447Z 
2022-05-18T04:01:19.9446779Z Generating XML reports...
2022-05-18T04:01:19.9480999Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040118.xml
2022-05-18T04:01:20.7706065Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuga02u82
2022-05-18T04:01:20.7706663Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuga02u82/_remote_module_non_scriptable.py
2022-05-18T04:01:21.0260843Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:01:21.0270367Z 
2022-05-18T04:01:21.0270646Z Running tests...
2022-05-18T04:01:21.0271308Z ----------------------------------------------------------------------
2022-05-18T04:01:21.3468394Z   test_duplicate_name (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17097
2022-05-18T04:01:21.3491199Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17098
2022-05-18T04:01:21.3515246Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17099
2022-05-18T04:01:21.3540335Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 17100
2022-05-18T04:01:21.9420284Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpng_tc1je
2022-05-18T04:01:21.9421095Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpng_tc1je/_remote_module_non_scriptable.py
2022-05-18T04:01:21.9573232Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph2bkdwb7
2022-05-18T04:01:21.9574566Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph2bkdwb7/_remote_module_non_scriptable.py
2022-05-18T04:01:22.0137716Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa8u2bbxj
2022-05-18T04:01:22.0138554Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa8u2bbxj/_remote_module_non_scriptable.py
2022-05-18T04:01:22.0178950Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp131abx9q
2022-05-18T04:01:22.0180746Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp131abx9q/_remote_module_non_scriptable.py
2022-05-18T04:01:22.1913098Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:01:22.2057487Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:01:22.2619224Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:01:22.2673001Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:01:22.5576819Z ok (1.530s)
2022-05-18T04:01:22.5577003Z 
2022-05-18T04:01:22.5577372Z ----------------------------------------------------------------------
2022-05-18T04:01:22.5577639Z Ran 1 test in 1.531s
2022-05-18T04:01:22.5577756Z 
2022-05-18T04:01:22.5577805Z OK
2022-05-18T04:01:22.5577897Z 
2022-05-18T04:01:22.5577996Z Generating XML reports...
2022-05-18T04:01:22.5611240Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040121.xml
2022-05-18T04:01:23.3124969Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx1mlty04
2022-05-18T04:01:23.3125674Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx1mlty04/_remote_module_non_scriptable.py
2022-05-18T04:01:23.5675164Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:01:23.5685178Z 
2022-05-18T04:01:23.5685257Z Running tests...
2022-05-18T04:01:23.5685659Z ----------------------------------------------------------------------
2022-05-18T04:01:23.8900920Z   test_duplicate_name_2 (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17228
2022-05-18T04:01:23.8923623Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17229
2022-05-18T04:01:23.8946839Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17230
2022-05-18T04:01:23.8971013Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 17231
2022-05-18T04:01:24.5115269Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdi6yooem
2022-05-18T04:01:24.5116487Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdi6yooem/_remote_module_non_scriptable.py
2022-05-18T04:01:24.5161482Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbw6dp0x5
2022-05-18T04:01:24.5162983Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbw6dp0x5/_remote_module_non_scriptable.py
2022-05-18T04:01:24.5207195Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8cra8szd
2022-05-18T04:01:24.5208560Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8cra8szd/_remote_module_non_scriptable.py
2022-05-18T04:01:24.5288217Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqyrr85n8
2022-05-18T04:01:24.5289719Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqyrr85n8/_remote_module_non_scriptable.py
2022-05-18T04:01:24.7584719Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:01:24.7627415Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:01:24.7680658Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:01:24.7776544Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:01:25.0005833Z ok (1.432s)
2022-05-18T04:01:25.0006088Z 
2022-05-18T04:01:25.0006614Z ----------------------------------------------------------------------
2022-05-18T04:01:25.0006977Z Ran 1 test in 1.432s
2022-05-18T04:01:25.0007095Z 
2022-05-18T04:01:25.0007156Z OK
2022-05-18T04:01:25.0007246Z 
2022-05-18T04:01:25.0007361Z Generating XML reports...
2022-05-18T04:01:25.0041215Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040123.xml
2022-05-18T04:01:25.7589020Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzvj_ojcd
2022-05-18T04:01:25.7589919Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzvj_ojcd/_remote_module_non_scriptable.py
2022-05-18T04:01:26.0134405Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:01:26.0143758Z 
2022-05-18T04:01:26.0143983Z Running tests...
2022-05-18T04:01:26.0144443Z ----------------------------------------------------------------------
2022-05-18T04:01:26.3347812Z   test_expected_src (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17359
2022-05-18T04:01:26.3369591Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17360
2022-05-18T04:01:26.3392659Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17361
2022-05-18T04:01:26.3416504Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 17362
2022-05-18T04:01:26.9667150Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj3pduo3u
2022-05-18T04:01:26.9668673Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj3pduo3u/_remote_module_non_scriptable.py
2022-05-18T04:01:26.9767024Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzk9b20py
2022-05-18T04:01:26.9767905Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzk9b20py/_remote_module_non_scriptable.py
2022-05-18T04:01:26.9778973Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp03n0bv3t
2022-05-18T04:01:26.9780341Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp03n0bv3t/_remote_module_non_scriptable.py
2022-05-18T04:01:27.0064115Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc7vqcj5o
2022-05-18T04:01:27.0065107Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc7vqcj5o/_remote_module_non_scriptable.py
2022-05-18T04:01:27.2153550Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:01:27.2251204Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:01:27.2260397Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:01:27.2547711Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:01:27.6454877Z ok (1.631s)
2022-05-18T04:01:27.6455134Z 
2022-05-18T04:01:27.6455654Z ----------------------------------------------------------------------
2022-05-18T04:01:27.6456073Z Ran 1 test in 1.631s
2022-05-18T04:01:27.6456199Z 
2022-05-18T04:01:27.6456262Z OK
2022-05-18T04:01:27.6456356Z 
2022-05-18T04:01:27.6456453Z Generating XML reports...
2022-05-18T04:01:27.6489416Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040126.xml
2022-05-18T04:01:28.4164113Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm9keem41
2022-05-18T04:01:28.4164790Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm9keem41/_remote_module_non_scriptable.py
2022-05-18T04:01:28.6687431Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:01:28.6697884Z 
2022-05-18T04:01:28.6697966Z Running tests...
2022-05-18T04:01:28.6698381Z ----------------------------------------------------------------------
2022-05-18T04:01:28.9849938Z   test_function_not_on_callee (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17578
2022-05-18T04:01:28.9872089Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17579
2022-05-18T04:01:28.9895210Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17580
2022-05-18T04:01:28.9919220Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 17581
2022-05-18T04:01:29.5744366Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyvmhonys
2022-05-18T04:01:29.5745613Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyvmhonys/_remote_module_non_scriptable.py
2022-05-18T04:01:29.6324807Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_75ywcva
2022-05-18T04:01:29.6325576Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjwg2fcn2
2022-05-18T04:01:29.6326341Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_75ywcva/_remote_module_non_scriptable.py
2022-05-18T04:01:29.6327098Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjwg2fcn2/_remote_module_non_scriptable.py
2022-05-18T04:01:29.6371705Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpullxbhgh
2022-05-18T04:01:29.6373170Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpullxbhgh/_remote_module_non_scriptable.py
2022-05-18T04:01:29.8212243Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:01:29.8803957Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:01:29.8805739Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:01:29.8841801Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:01:30.1228266Z On WorkerInfo(id=1, name=worker1):
2022-05-18T04:01:30.1232249Z AttributeError("Can't get attribute 'foo_add' on <module 'torch.testing._internal.distributed.rpc.rpc_test' from '/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py'> Default RPC pickler does not serialize\n            function code. Ensure that UDFs are defined on both caller and\n            callee modules.")
2022-05-18T04:01:30.1233855Z Traceback (most recent call last):
2022-05-18T04:01:30.1234816Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 159, in deserialize
2022-05-18T04:01:30.1235445Z     ret = unpickler.load()
2022-05-18T04:01:30.1236660Z AttributeError: Can't get attribute 'foo_add' on <module 'torch.testing._internal.distributed.rpc.rpc_test' from '/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py'>
2022-05-18T04:01:30.1237336Z 
2022-05-18T04:01:30.1237632Z The above exception was the direct cause of the following exception:
2022-05-18T04:01:30.1237974Z 
2022-05-18T04:01:30.1238174Z Traceback (most recent call last):
2022-05-18T04:01:30.1239011Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 205, in _run_function
2022-05-18T04:01:30.1239601Z     raise python_udf
2022-05-18T04:01:30.1240647Z AttributeError: Can't get attribute 'foo_add' on <module 'torch.testing._internal.distributed.rpc.rpc_test' from '/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py'> Default RPC pickler does not serialize
2022-05-18T04:01:30.1241690Z             function code. Ensure that UDFs are defined on both caller and
2022-05-18T04:01:30.1242188Z             callee modules.
2022-05-18T04:01:30.1242429Z 
2022-05-18T04:01:30.3959576Z ok (1.726s)
2022-05-18T04:01:30.3959727Z 
2022-05-18T04:01:30.3960087Z ----------------------------------------------------------------------
2022-05-18T04:01:30.3960340Z Ran 1 test in 1.726s
2022-05-18T04:01:30.3960458Z 
2022-05-18T04:01:30.3960520Z OK
2022-05-18T04:01:30.3960640Z 
2022-05-18T04:01:30.3960769Z Generating XML reports...
2022-05-18T04:01:30.3995139Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040128.xml
2022-05-18T04:01:31.2015118Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjduff488
2022-05-18T04:01:31.2016039Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjduff488/_remote_module_non_scriptable.py
2022-05-18T04:01:31.4590749Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:01:31.4600717Z 
2022-05-18T04:01:31.4601192Z Running tests...
2022-05-18T04:01:31.4601636Z ----------------------------------------------------------------------
2022-05-18T04:01:31.7846912Z   test_future_done (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17797
2022-05-18T04:01:31.7870407Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17798
2022-05-18T04:01:31.7893836Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17799
2022-05-18T04:01:31.7917695Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 17800
2022-05-18T04:01:32.4450559Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvt49hwx5
2022-05-18T04:01:32.4451476Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvt49hwx5/_remote_module_non_scriptable.py
2022-05-18T04:01:32.4538870Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm0fh8g5_
2022-05-18T04:01:32.4539743Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm0fh8g5_/_remote_module_non_scriptable.py
2022-05-18T04:01:32.4844399Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb2orp6kr
2022-05-18T04:01:32.4845403Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb2orp6kr/_remote_module_non_scriptable.py
2022-05-18T04:01:32.5031721Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb0lcu3iz
2022-05-18T04:01:32.5032929Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb0lcu3iz/_remote_module_non_scriptable.py
2022-05-18T04:01:32.6970503Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:01:32.7056321Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:01:32.7368218Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:01:32.7531433Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:01:33.1959014Z ok (1.736s)
2022-05-18T04:01:33.1959192Z 
2022-05-18T04:01:33.1959671Z ----------------------------------------------------------------------
2022-05-18T04:01:33.1960140Z Ran 1 test in 1.736s
2022-05-18T04:01:33.1960329Z 
2022-05-18T04:01:33.1960389Z OK
2022-05-18T04:01:33.1960482Z 
2022-05-18T04:01:33.1960575Z Generating XML reports...
2022-05-18T04:01:33.1996056Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040131.xml
2022-05-18T04:01:33.9764286Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzhpu71_q
2022-05-18T04:01:33.9765035Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzhpu71_q/_remote_module_non_scriptable.py
2022-05-18T04:01:34.2289120Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:01:34.2298825Z 
2022-05-18T04:01:34.2299092Z Running tests...
2022-05-18T04:01:34.2299611Z ----------------------------------------------------------------------
2022-05-18T04:01:34.5459335Z   test_future_done_exception (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18016
2022-05-18T04:01:34.5482566Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18017
2022-05-18T04:01:34.5506151Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18018
2022-05-18T04:01:34.5530287Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 18019
2022-05-18T04:01:35.1919702Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnconvow_
2022-05-18T04:01:35.1920729Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnconvow_/_remote_module_non_scriptable.py
2022-05-18T04:01:35.2658284Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsh6cxp5y
2022-05-18T04:01:35.2659187Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsh6cxp5y/_remote_module_non_scriptable.py
2022-05-18T04:01:35.2851902Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeykgk3lg
2022-05-18T04:01:35.2853140Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeykgk3lg/_remote_module_non_scriptable.py
2022-05-18T04:01:35.2863355Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8sd8zdon
2022-05-18T04:01:35.2865320Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8sd8zdon/_remote_module_non_scriptable.py
2022-05-18T04:01:35.4380532Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:01:35.5138162Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:01:35.5325722Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:01:35.5327291Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:01:35.7487649Z On WorkerInfo(id=1, name=worker1):
2022-05-18T04:01:35.7488383Z ValueError('Expected error')
2022-05-18T04:01:35.7488852Z Traceback (most recent call last):
2022-05-18T04:01:35.7489652Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function
2022-05-18T04:01:35.7490307Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-05-18T04:01:35.7491239Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 396, in raise_func
2022-05-18T04:01:35.7492156Z     raise ValueError(expected_err)
2022-05-18T04:01:35.7492704Z ValueError: Expected error
2022-05-18T04:01:35.7492865Z 
2022-05-18T04:01:35.7644835Z On WorkerInfo(id=2, name=worker2):
2022-05-18T04:01:35.7645876Z ValueError('Expected error')
2022-05-18T04:01:35.7646351Z Traceback (most recent call last):
2022-05-18T04:01:35.7647238Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function
2022-05-18T04:01:35.7647967Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-05-18T04:01:35.7648952Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 396, in raise_func
2022-05-18T04:01:35.7649610Z     raise ValueError(expected_err)
2022-05-18T04:01:35.7650043Z ValueError: Expected error
2022-05-18T04:01:35.7650313Z 
2022-05-18T04:01:35.7686101Z On WorkerInfo(id=0, name=worker0):
2022-05-18T04:01:35.7686618Z ValueError('Expected error')
2022-05-18T04:01:35.7686972Z Traceback (most recent call last):
2022-05-18T04:01:35.7687680Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function
2022-05-18T04:01:35.7688267Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-05-18T04:01:35.7689040Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 396, in raise_func
2022-05-18T04:01:35.7689550Z     raise ValueError(expected_err)
2022-05-18T04:01:35.7689921Z ValueError: Expected error
2022-05-18T04:01:35.7690137Z 
2022-05-18T04:01:35.7772632Z On WorkerInfo(id=3, name=worker3):
2022-05-18T04:01:35.7773292Z ValueError('Expected error')
2022-05-18T04:01:35.7775645Z Traceback (most recent call last):
2022-05-18T04:01:35.7776499Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function
2022-05-18T04:01:35.7776877Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-05-18T04:01:35.7777325Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 396, in raise_func
2022-05-18T04:01:35.7777654Z     raise ValueError(expected_err)
2022-05-18T04:01:35.7777875Z ValueError: Expected error
2022-05-18T04:01:35.7778000Z 
2022-05-18T04:01:35.9570803Z ok (1.727s)
2022-05-18T04:01:35.9571027Z 
2022-05-18T04:01:35.9571496Z ----------------------------------------------------------------------
2022-05-18T04:01:35.9571882Z Ran 1 test in 1.727s
2022-05-18T04:01:35.9572057Z 
2022-05-18T04:01:35.9572156Z OK
2022-05-18T04:01:35.9572304Z 
2022-05-18T04:01:35.9572498Z Generating XML reports...
2022-05-18T04:01:35.9606896Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040134.xml
2022-05-18T04:01:36.7632226Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbnjfuppy
2022-05-18T04:01:36.7632700Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbnjfuppy/_remote_module_non_scriptable.py
2022-05-18T04:01:37.0149934Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:01:37.0159643Z 
2022-05-18T04:01:37.0159732Z Running tests...
2022-05-18T04:01:37.0160451Z ----------------------------------------------------------------------
2022-05-18T04:01:37.3379535Z   test_future_in_rpc (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18235
2022-05-18T04:01:37.3402204Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18236
2022-05-18T04:01:37.3425197Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18237
2022-05-18T04:01:37.3450239Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 18238
2022-05-18T04:01:37.9953783Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkog6az9d
2022-05-18T04:01:37.9954871Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkog6az9d/_remote_module_non_scriptable.py
2022-05-18T04:01:38.0534958Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmy_f2mk9
2022-05-18T04:01:38.0535970Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmy_f2mk9/_remote_module_non_scriptable.py
2022-05-18T04:01:38.0598989Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq2i86ram
2022-05-18T04:01:38.0600372Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq2i86ram/_remote_module_non_scriptable.py
2022-05-18T04:01:38.0639380Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_2joy61a
2022-05-18T04:01:38.0641331Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_2joy61a/_remote_module_non_scriptable.py
2022-05-18T04:01:38.2465505Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:01:38.3008968Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:01:38.3073456Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:01:38.3108842Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:01:38.7490007Z ok (1.733s)
2022-05-18T04:01:38.7490245Z 
2022-05-18T04:01:38.7490754Z ----------------------------------------------------------------------
2022-05-18T04:01:38.7491123Z Ran 1 test in 1.733s
2022-05-18T04:01:38.7491239Z 
2022-05-18T04:01:38.7491300Z OK
2022-05-18T04:01:38.7491392Z 
2022-05-18T04:01:38.7491485Z Generating XML reports...
2022-05-18T04:01:38.7527499Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040137.xml
2022-05-18T04:01:39.5185967Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfv30w460
2022-05-18T04:01:39.5187084Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfv30w460/_remote_module_non_scriptable.py
2022-05-18T04:01:39.7719026Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:01:39.7728980Z 
2022-05-18T04:01:39.7729258Z Running tests...
2022-05-18T04:01:39.7729952Z ----------------------------------------------------------------------
2022-05-18T04:01:40.0883918Z   test_future_nested_callback (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18454
2022-05-18T04:01:40.0906636Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18455
2022-05-18T04:01:40.0929680Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18456
2022-05-18T04:01:40.0953938Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 18457
2022-05-18T04:01:40.7734816Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvzhqhbdk
2022-05-18T04:01:40.7736106Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvzhqhbdk/_remote_module_non_scriptable.py
2022-05-18T04:01:40.7957974Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz5l1muok
2022-05-18T04:01:40.7958778Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz5l1muok/_remote_module_non_scriptable.py
2022-05-18T04:01:40.8314952Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpctu50u43
2022-05-18T04:01:40.8315659Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpctu50u43/_remote_module_non_scriptable.py
2022-05-18T04:01:40.8451824Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcb3uyzzd
2022-05-18T04:01:40.8452872Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcb3uyzzd/_remote_module_non_scriptable.py
2022-05-18T04:01:41.0221114Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:01:41.0435654Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:01:41.0790167Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:01:41.0911847Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:01:41.4993562Z ok (1.726s)
2022-05-18T04:01:41.4993774Z 
2022-05-18T04:01:41.4994149Z ----------------------------------------------------------------------
2022-05-18T04:01:41.4994445Z Ran 1 test in 1.726s
2022-05-18T04:01:41.4994561Z 
2022-05-18T04:01:41.4994623Z OK
2022-05-18T04:01:41.4994717Z 
2022-05-18T04:01:41.4994812Z Generating XML reports...
2022-05-18T04:01:41.5028677Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040139.xml
2022-05-18T04:01:42.2758715Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuqprixgy
2022-05-18T04:01:42.2759601Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuqprixgy/_remote_module_non_scriptable.py
2022-05-18T04:01:42.5291665Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:01:42.5301458Z 
2022-05-18T04:01:42.5301557Z Running tests...
2022-05-18T04:01:42.5302090Z ----------------------------------------------------------------------
2022-05-18T04:01:42.8365269Z   test_future_wait_twice (__main__.TensorPipeRpcTest) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/69480 for allplatform(s) . If you're seeing this on your local machine and would like to enable this test, please make sure IN_CI is not set and you are not using the flag --import-disabled-tests. (0.306s)
2022-05-18T04:01:42.8365751Z 
2022-05-18T04:01:42.8365975Z ----------------------------------------------------------------------
2022-05-18T04:01:42.8366224Z Ran 1 test in 0.306s
2022-05-18T04:01:42.8366325Z 
2022-05-18T04:01:42.8366398Z OK (skipped=1)
2022-05-18T04:01:42.8366506Z 
2022-05-18T04:01:42.8366596Z Generating XML reports...
2022-05-18T04:01:42.8388553Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040142.xml
2022-05-18T04:01:43.5455576Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplf1vb25t
2022-05-18T04:01:43.5456679Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplf1vb25t/_remote_module_non_scriptable.py
2022-05-18T04:01:43.7972784Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:01:43.7982407Z 
2022-05-18T04:01:43.7982695Z Running tests...
2022-05-18T04:01:43.7983400Z ----------------------------------------------------------------------
2022-05-18T04:01:44.1110690Z   test_get_worker_infos (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18683
2022-05-18T04:01:44.1132779Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18684
2022-05-18T04:01:44.1155836Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18685
2022-05-18T04:01:44.1179702Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 18686
2022-05-18T04:01:44.7864442Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7iq_fy24
2022-05-18T04:01:44.7865621Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7iq_fy24/_remote_module_non_scriptable.py
2022-05-18T04:01:44.8005647Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgadooqg2
2022-05-18T04:01:44.8006618Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgadooqg2/_remote_module_non_scriptable.py
2022-05-18T04:01:44.8026697Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8zb6vfod
2022-05-18T04:01:44.8028365Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8zb6vfod/_remote_module_non_scriptable.py
2022-05-18T04:01:44.8181910Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7m3d_4wz
2022-05-18T04:01:44.8183157Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7m3d_4wz/_remote_module_non_scriptable.py
2022-05-18T04:01:45.0359257Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:01:45.0503574Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:01:45.0508988Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:01:45.0699346Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:01:45.5220426Z ok (1.723s)
2022-05-18T04:01:45.5220704Z 
2022-05-18T04:01:45.5221137Z ----------------------------------------------------------------------
2022-05-18T04:01:45.5221379Z Ran 1 test in 1.724s
2022-05-18T04:01:45.5221494Z 
2022-05-18T04:01:45.5221560Z OK
2022-05-18T04:01:45.5221652Z 
2022-05-18T04:01:45.5221748Z Generating XML reports...
2022-05-18T04:01:45.5255475Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040143.xml
2022-05-18T04:01:46.2992205Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9tnzbf4o
2022-05-18T04:01:46.2992983Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9tnzbf4o/_remote_module_non_scriptable.py
2022-05-18T04:01:46.5517336Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:01:46.5526851Z 
2022-05-18T04:01:46.5527124Z Running tests...
2022-05-18T04:01:46.5527501Z ----------------------------------------------------------------------
2022-05-18T04:01:46.5530996Z   test_graceful_shutdown_with_uneven_workload (__main__.TensorPipeRpcTest)
2022-05-18T04:01:46.8766819Z Test graceful termination. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18902
2022-05-18T04:01:46.8790911Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18903
2022-05-18T04:01:46.8814628Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18904
2022-05-18T04:01:46.8840083Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 18905
2022-05-18T04:01:47.5179590Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptfpw0aor
2022-05-18T04:01:47.5180569Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptfpw0aor/_remote_module_non_scriptable.py
2022-05-18T04:01:47.5401640Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6x2lbld5
2022-05-18T04:01:47.5404476Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6x2lbld5/_remote_module_non_scriptable.py
2022-05-18T04:01:47.5761290Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9coajewl
2022-05-18T04:01:47.5762550Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9coajewl/_remote_module_non_scriptable.py
2022-05-18T04:01:47.6092374Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp206yuixx
2022-05-18T04:01:47.6093323Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp206yuixx/_remote_module_non_scriptable.py
2022-05-18T04:01:47.7664144Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:01:47.7884216Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:01:47.8248565Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:01:47.8573259Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:01:48.4881097Z ok (1.935s)
2022-05-18T04:01:48.4881310Z 
2022-05-18T04:01:48.4881623Z ----------------------------------------------------------------------
2022-05-18T04:01:48.4881898Z Ran 1 test in 1.935s
2022-05-18T04:01:48.4882018Z 
2022-05-18T04:01:48.4882079Z OK
2022-05-18T04:01:48.4882400Z 
2022-05-18T04:01:48.4882496Z Generating XML reports...
2022-05-18T04:01:48.4918965Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040146.xml
2022-05-18T04:01:49.2586787Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsv2o8zzn
2022-05-18T04:01:49.2587402Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsv2o8zzn/_remote_module_non_scriptable.py
2022-05-18T04:01:49.5106813Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:01:49.5116478Z 
2022-05-18T04:01:49.5116617Z Running tests...
2022-05-18T04:01:49.5117134Z ----------------------------------------------------------------------
2022-05-18T04:01:49.8275458Z   test_handle_send_exceptions (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19121
2022-05-18T04:01:49.8297394Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19122
2022-05-18T04:01:49.8320343Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19123
2022-05-18T04:01:49.8345462Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 19124
2022-05-18T04:01:50.4636579Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzdb0p4hn
2022-05-18T04:01:50.4637525Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzdb0p4hn/_remote_module_non_scriptable.py
2022-05-18T04:01:50.4707280Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb1kw8_2b
2022-05-18T04:01:50.4708787Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb1kw8_2b/_remote_module_non_scriptable.py
2022-05-18T04:01:50.4748600Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4ywzouax
2022-05-18T04:01:50.4749898Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4ywzouax/_remote_module_non_scriptable.py
2022-05-18T04:01:50.4755414Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphtqhkom7
2022-05-18T04:01:50.4757277Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphtqhkom7/_remote_module_non_scriptable.py
2022-05-18T04:01:50.7129247Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:01:50.7171340Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:01:50.7221716Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:01:50.7226627Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:01:50.9510409Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:01:50.9609041Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:01:50.9714100Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T04:01:50.9714925Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:01:50.9716772Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:01:50.9718794Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:01:50.9720129Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:01:50.9721275Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:01:50.9820435Z [W tensorpipe_agent.cpp:728] RPC agent for worker0 encountered error when reading incoming request from worker1: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2022-05-18T04:01:50.9825358Z [W tensorpipe_agent.cpp:728] RPC agent for worker2 encountered error when reading incoming request from worker0: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2022-05-18T04:01:50.9826582Z [W tensorpipe_agent.cpp:728] RPC agent for worker1 encountered error when reading incoming request from worker0: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2022-05-18T04:01:50.9827769Z [W tensorpipe_agent.cpp:728] RPC agent for worker0 encountered error when reading incoming request from worker2: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2022-05-18T04:01:50.9828798Z [W tensorpipe_agent.cpp:728] RPC agent for worker0 encountered error when reading incoming request from worker3: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2022-05-18T04:01:50.9829934Z [W tensorpipe_agent.cpp:728] RPC agent for worker2 encountered error when reading incoming request from worker1: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2022-05-18T04:01:50.9830982Z [W tensorpipe_agent.cpp:728] RPC agent for worker3 encountered error when reading incoming request from worker0: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2022-05-18T04:01:50.9841916Z [W tensorpipe_agent.cpp:918] RPC agent for worker1 encountered error when sending outgoing request #1 to worker2: connect: Connection refused (this error originated at tensorpipe/common/socket.cc:114)
2022-05-18T04:01:50.9847584Z [W tensorpipe_agent.cpp:918] RPC agent for worker1 encountered error when sending outgoing request #2 to worker2: connect: Connection refused (this error originated at tensorpipe/common/socket.cc:114)
2022-05-18T04:01:51.2385008Z ok (1.727s)
2022-05-18T04:01:51.2385281Z 
2022-05-18T04:01:51.2385731Z ----------------------------------------------------------------------
2022-05-18T04:01:51.2386128Z Ran 1 test in 1.727s
2022-05-18T04:01:51.2386314Z 
2022-05-18T04:01:51.2386425Z OK
2022-05-18T04:01:51.2386557Z 
2022-05-18T04:01:51.2386694Z Generating XML reports...
2022-05-18T04:01:51.2421245Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040149.xml
2022-05-18T04:01:52.0042430Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvgqm8ivu
2022-05-18T04:01:52.0043140Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvgqm8ivu/_remote_module_non_scriptable.py
2022-05-18T04:01:52.2580557Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:01:52.2590245Z 
2022-05-18T04:01:52.2590435Z Running tests...
2022-05-18T04:01:52.2590867Z ----------------------------------------------------------------------
2022-05-18T04:01:52.5785864Z   test_ignore_rref_leak (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19352
2022-05-18T04:01:52.5807984Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19353
2022-05-18T04:01:52.5831105Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19354
2022-05-18T04:01:52.5855106Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 19355
2022-05-18T04:01:53.1923527Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2qa8l4ye
2022-05-18T04:01:53.1924334Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2qa8l4ye/_remote_module_non_scriptable.py
2022-05-18T04:01:53.2154473Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq96dsoy1
2022-05-18T04:01:53.2155491Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq96dsoy1/_remote_module_non_scriptable.py
2022-05-18T04:01:53.2371647Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv3pm6ln6
2022-05-18T04:01:53.2372476Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxzi4tixy
2022-05-18T04:01:53.2373470Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv3pm6ln6/_remote_module_non_scriptable.py
2022-05-18T04:01:53.2374309Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxzi4tixy/_remote_module_non_scriptable.py
2022-05-18T04:01:53.4436692Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:01:53.4642712Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:01:53.4836691Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:01:53.4849599Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:01:53.7232200Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:01:53.7332211Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:01:53.7434663Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:01:53.7436070Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T04:01:53.7438588Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:01:53.7439738Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:01:53.7440884Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:01:53.7441996Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:01:53.7842519Z [W rref_context.cpp:156] Detected RRef Leaks during shutdown. This usually occurs when the application code still holds references to RRef instances when calling shutdown(). If the program has completed correctly and the process is exiting, it is OK to ignore these leaks. However, if you program will keep running after this, these leaks could result in memory leaks on RRef owners. Please make sure all RRefs are out of scope and Python GC has deleted them before calling shutdown(): 
2022-05-18T04:01:53.7844115Z Leaking RRef GloballyUniqueId(created_on=3, local_id=0) with fork Id GloballyUniqueId(created_on=3, local_id=1)
2022-05-18T04:01:53.7844657Z 
2022-05-18T04:01:53.7862709Z [W rref_context.cpp:156] Detected RRef Leaks during shutdown. This usually occurs when the application code still holds references to RRef instances when calling shutdown(). If the program has completed correctly and the process is exiting, it is OK to ignore these leaks. However, if you program will keep running after this, these leaks could result in memory leaks on RRef owners. Please make sure all RRefs are out of scope and Python GC has deleted them before calling shutdown(): 
2022-05-18T04:01:53.7864584Z Leaking RRef GloballyUniqueId(created_on=1, local_id=0) with fork Id GloballyUniqueId(created_on=1, local_id=1)
2022-05-18T04:01:53.7865142Z 
2022-05-18T04:01:53.7869425Z [W rref_context.cpp:156] Detected RRef Leaks during shutdown. This usually occurs when the application code still holds references to RRef instances when calling shutdown(). If the program has completed correctly and the process is exiting, it is OK to ignore these leaks. However, if you program will keep running after this, these leaks could result in memory leaks on RRef owners. Please make sure all RRefs are out of scope and Python GC has deleted them before calling shutdown(): 
2022-05-18T04:01:53.7871294Z Leaking RRef GloballyUniqueId(created_on=0, local_id=0) with fork Id GloballyUniqueId(created_on=0, local_id=1)
2022-05-18T04:01:53.7871531Z 
2022-05-18T04:01:53.7929626Z [W rref_context.cpp:156] Detected RRef Leaks during shutdown. This usually occurs when the application code still holds references to RRef instances when calling shutdown(). If the program has completed correctly and the process is exiting, it is OK to ignore these leaks. However, if you program will keep running after this, these leaks could result in memory leaks on RRef owners. Please make sure all RRefs are out of scope and Python GC has deleted them before calling shutdown(): 
2022-05-18T04:01:53.7931028Z Leaking RRef GloballyUniqueId(created_on=2, local_id=0) with fork Id GloballyUniqueId(created_on=2, local_id=1)
2022-05-18T04:01:53.7931388Z 
2022-05-18T04:01:53.9895016Z ok (1.730s)
2022-05-18T04:01:53.9895300Z 
2022-05-18T04:01:53.9895780Z ----------------------------------------------------------------------
2022-05-18T04:01:53.9896039Z Ran 1 test in 1.730s
2022-05-18T04:01:53.9896160Z 
2022-05-18T04:01:53.9896223Z OK
2022-05-18T04:01:53.9896316Z 
2022-05-18T04:01:53.9896397Z Generating XML reports...
2022-05-18T04:01:53.9929565Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040152.xml
2022-05-18T04:01:54.7580217Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpowq5z2ns
2022-05-18T04:01:54.7580701Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpowq5z2ns/_remote_module_non_scriptable.py
2022-05-18T04:01:55.0093043Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:01:55.0103219Z 
2022-05-18T04:01:55.0103325Z Running tests...
2022-05-18T04:01:55.0103904Z ----------------------------------------------------------------------
2022-05-18T04:01:55.3294685Z   test_init_dynamic_and_static_rpc_group (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19583
2022-05-18T04:01:55.3317503Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19584
2022-05-18T04:01:55.3340836Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19585
2022-05-18T04:01:55.3365534Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 19586
2022-05-18T04:01:55.9175912Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpewmqtmkj
2022-05-18T04:01:55.9176637Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpewmqtmkj/_remote_module_non_scriptable.py
2022-05-18T04:01:55.9613153Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2attm342
2022-05-18T04:01:55.9614502Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2attm342/_remote_module_non_scriptable.py
2022-05-18T04:01:55.9938629Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxgzo3d3u
2022-05-18T04:01:55.9939391Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxgzo3d3u/_remote_module_non_scriptable.py
2022-05-18T04:01:55.9990627Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps7n1k1hi
2022-05-18T04:01:55.9992365Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps7n1k1hi/_remote_module_non_scriptable.py
2022-05-18T04:01:56.1659010Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:01:56.2080667Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:01:56.2404018Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:01:56.2467412Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:01:56.2672611Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:01:56.2773343Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:01:56.2779686Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:01:56.2780102Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T04:01:56.2780889Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:01:56.2781485Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:01:56.2876564Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:01:56.2877430Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:01:56.6430298Z [W tensorpipe_agent.cpp:728] RPC agent for worker0 encountered error when reading incoming request from worker2: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2022-05-18T04:01:56.6724813Z [W tensorpipe_agent.cpp:728] RPC agent for worker1 encountered error when reading incoming request from worker0: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2022-05-18T04:01:56.7404743Z ok (1.730s)
2022-05-18T04:01:56.7405001Z 
2022-05-18T04:01:56.7405414Z ----------------------------------------------------------------------
2022-05-18T04:01:56.7405670Z Ran 1 test in 1.730s
2022-05-18T04:01:56.7405790Z 
2022-05-18T04:01:56.7405854Z OK
2022-05-18T04:01:56.7405933Z 
2022-05-18T04:01:56.7406026Z Generating XML reports...
2022-05-18T04:01:56.7440159Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040155.xml
2022-05-18T04:01:57.5113118Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt1311c5s
2022-05-18T04:01:57.5113841Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt1311c5s/_remote_module_non_scriptable.py
2022-05-18T04:01:57.7677606Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:01:57.7687424Z 
2022-05-18T04:01:57.7687905Z Running tests...
2022-05-18T04:01:57.7688307Z ----------------------------------------------------------------------
2022-05-18T04:01:58.0848858Z   test_init_pg_then_rpc (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19789
2022-05-18T04:01:58.0872291Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19790
2022-05-18T04:01:58.0895813Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19791
2022-05-18T04:01:58.0919722Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 19792
2022-05-18T04:01:58.7782860Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo2rfbe99
2022-05-18T04:01:58.7784230Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo2rfbe99/_remote_module_non_scriptable.py
2022-05-18T04:01:58.8257891Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp94e8f3ss
2022-05-18T04:01:58.8258726Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp94e8f3ss/_remote_module_non_scriptable.py
2022-05-18T04:01:58.8383568Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz255jedo
2022-05-18T04:01:58.8384886Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz255jedo/_remote_module_non_scriptable.py
2022-05-18T04:01:58.8453529Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4gyqb7de
2022-05-18T04:01:58.8454951Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4gyqb7de/_remote_module_non_scriptable.py
2022-05-18T04:01:59.0268826Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:01:59.0732884Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:01:59.0835738Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:01:59.0921229Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:01:59.1144812Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:01:59.1145338Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:01:59.1245318Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:01:59.1246094Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T04:01:59.1246838Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:01:59.1247550Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:01:59.1248078Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:01:59.1248607Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:01:59.5960423Z ok (1.827s)
2022-05-18T04:01:59.5960689Z 
2022-05-18T04:01:59.5961007Z ----------------------------------------------------------------------
2022-05-18T04:01:59.5961248Z Ran 1 test in 1.827s
2022-05-18T04:01:59.5961364Z 
2022-05-18T04:01:59.5961428Z OK
2022-05-18T04:01:59.5961520Z 
2022-05-18T04:01:59.5961615Z Generating XML reports...
2022-05-18T04:01:59.5994758Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040157.xml
2022-05-18T04:02:00.3619451Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3h8sg9c1
2022-05-18T04:02:00.3620125Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3h8sg9c1/_remote_module_non_scriptable.py
2022-05-18T04:02:00.6151566Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:02:00.6160986Z 
2022-05-18T04:02:00.6161106Z Running tests...
2022-05-18T04:02:00.6161662Z ----------------------------------------------------------------------
2022-05-18T04:02:00.9293377Z   test_init_rpc_then_pg (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20020
2022-05-18T04:02:00.9315222Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20021
2022-05-18T04:02:00.9338123Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20022
2022-05-18T04:02:00.9363227Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 20023
2022-05-18T04:02:01.5331057Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm55bdo4e
2022-05-18T04:02:01.5332480Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm55bdo4e/_remote_module_non_scriptable.py
2022-05-18T04:02:01.5366069Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt90mhfmj
2022-05-18T04:02:01.5367210Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt90mhfmj/_remote_module_non_scriptable.py
2022-05-18T04:02:01.5529715Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpex2cw7ro
2022-05-18T04:02:01.5531009Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpex2cw7ro/_remote_module_non_scriptable.py
2022-05-18T04:02:01.5812769Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph814wi9a
2022-05-18T04:02:01.5813459Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph814wi9a/_remote_module_non_scriptable.py
2022-05-18T04:02:01.7783322Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:02:01.7835989Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:02:01.8040035Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:02:01.8326005Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:02:02.0771463Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:02:02.0771917Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:02:02.0875571Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T04:02:02.0876496Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:02:02.0877156Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:02:02.0877870Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:02:02.0878619Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:02:02.0879459Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:02:02.3402150Z ok (1.724s)
2022-05-18T04:02:02.3402627Z 
2022-05-18T04:02:02.3403336Z ----------------------------------------------------------------------
2022-05-18T04:02:02.3403775Z Ran 1 test in 1.724s
2022-05-18T04:02:02.3403951Z 
2022-05-18T04:02:02.3404046Z OK
2022-05-18T04:02:02.3404148Z 
2022-05-18T04:02:02.3404246Z Generating XML reports...
2022-05-18T04:02:02.3437742Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040200.xml
2022-05-18T04:02:03.1117220Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptubwl6tj
2022-05-18T04:02:03.1117975Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptubwl6tj/_remote_module_non_scriptable.py
2022-05-18T04:02:03.3661755Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:02:03.3671814Z 
2022-05-18T04:02:03.3672293Z Running tests...
2022-05-18T04:02:03.3672699Z ----------------------------------------------------------------------
2022-05-18T04:02:03.6815025Z   test_init_rpc_twice (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20251
2022-05-18T04:02:03.6837448Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20252
2022-05-18T04:02:03.6860651Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20253
2022-05-18T04:02:03.6884463Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 20254
2022-05-18T04:02:04.2957378Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa23ndcsw
2022-05-18T04:02:04.2958154Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa23ndcsw/_remote_module_non_scriptable.py
2022-05-18T04:02:04.3030619Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1l6xrei_
2022-05-18T04:02:04.3032646Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1l6xrei_/_remote_module_non_scriptable.py
2022-05-18T04:02:04.3144410Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpti7bo8cf
2022-05-18T04:02:04.3145704Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpti7bo8cf/_remote_module_non_scriptable.py
2022-05-18T04:02:04.3149336Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplex1l_et
2022-05-18T04:02:04.3151758Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplex1l_et/_remote_module_non_scriptable.py
2022-05-18T04:02:04.5445233Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:02:04.5509189Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:02:04.5618072Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:02:04.5629168Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:02:04.5818502Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:02:04.6022556Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:02:04.6023326Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:02:04.6023870Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T04:02:04.6024833Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:02:04.6025654Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:02:04.6026594Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:02:04.6027614Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:02:05.2927668Z ok (1.925s)
2022-05-18T04:02:05.2927930Z 
2022-05-18T04:02:05.2928449Z ----------------------------------------------------------------------
2022-05-18T04:02:05.2928879Z Ran 1 test in 1.926s
2022-05-18T04:02:05.2929000Z 
2022-05-18T04:02:05.2929060Z OK
2022-05-18T04:02:05.2929149Z 
2022-05-18T04:02:05.2929246Z Generating XML reports...
2022-05-18T04:02:05.2962414Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040203.xml
2022-05-18T04:02:06.0695100Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd6fn44sg
2022-05-18T04:02:06.0695888Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd6fn44sg/_remote_module_non_scriptable.py
2022-05-18T04:02:06.3277125Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:02:06.3287019Z 
2022-05-18T04:02:06.3287337Z Running tests...
2022-05-18T04:02:06.3287982Z ----------------------------------------------------------------------
2022-05-18T04:02:06.6468282Z   test_init_rpc_without_world_size (__main__.TensorPipeRpcTest) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/76511 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure IN_CI is not set and you are not using the flag --import-disabled-tests. (0.318s)
2022-05-18T04:02:06.6469273Z 
2022-05-18T04:02:06.6469628Z ----------------------------------------------------------------------
2022-05-18T04:02:06.6470025Z Ran 1 test in 0.318s
2022-05-18T04:02:06.6470241Z 
2022-05-18T04:02:06.6470353Z OK (skipped=1)
2022-05-18T04:02:06.6470528Z 
2022-05-18T04:02:06.6470670Z Generating XML reports...
2022-05-18T04:02:06.6493686Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040206.xml
2022-05-18T04:02:07.3782501Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbzhn7fnl
2022-05-18T04:02:07.3783758Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbzhn7fnl/_remote_module_non_scriptable.py
2022-05-18T04:02:07.6317853Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:02:07.6327269Z 
2022-05-18T04:02:07.6327433Z Running tests...
2022-05-18T04:02:07.6327778Z ----------------------------------------------------------------------
2022-05-18T04:02:07.9498864Z   test_init_rpc_without_world_size_without_rank (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20656
2022-05-18T04:02:07.9522424Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20657
2022-05-18T04:02:07.9545961Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20658
2022-05-18T04:02:07.9570481Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 20659
2022-05-18T04:02:08.6258503Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpin1uzffg
2022-05-18T04:02:08.6259516Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpin1uzffg/_remote_module_non_scriptable.py
2022-05-18T04:02:08.6403804Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqia9dae5
2022-05-18T04:02:08.6405203Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqia9dae5/_remote_module_non_scriptable.py
2022-05-18T04:02:08.6971966Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm8t04phg
2022-05-18T04:02:08.6972837Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm8t04phg/_remote_module_non_scriptable.py
2022-05-18T04:02:08.6977993Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptvj8coo1
2022-05-18T04:02:08.6980284Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptvj8coo1/_remote_module_non_scriptable.py
2022-05-18T04:02:08.8790074Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:02:08.8966096Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:02:08.9539446Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:02:08.9540145Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:02:09.1607605Z ok (1.528s)
2022-05-18T04:02:09.1607858Z 
2022-05-18T04:02:09.1608358Z ----------------------------------------------------------------------
2022-05-18T04:02:09.1608678Z Ran 1 test in 1.528s
2022-05-18T04:02:09.1608825Z 
2022-05-18T04:02:09.1608888Z OK
2022-05-18T04:02:09.1608979Z 
2022-05-18T04:02:09.1609059Z Generating XML reports...
2022-05-18T04:02:09.1645095Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040207.xml
2022-05-18T04:02:09.9288248Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo8hkpjs2
2022-05-18T04:02:09.9288799Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo8hkpjs2/_remote_module_non_scriptable.py
2022-05-18T04:02:10.1850644Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:02:10.1860511Z 
2022-05-18T04:02:10.1860601Z Running tests...
2022-05-18T04:02:10.1861164Z ----------------------------------------------------------------------
2022-05-18T04:02:10.5027381Z   test_int_callee (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20711
2022-05-18T04:02:10.5050613Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20712
2022-05-18T04:02:10.5073586Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20713
2022-05-18T04:02:10.5098030Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 20714
2022-05-18T04:02:11.1168192Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeemwwpsw
2022-05-18T04:02:11.1168948Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeemwwpsw/_remote_module_non_scriptable.py
2022-05-18T04:02:11.1486058Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp1ztfdi7
2022-05-18T04:02:11.1486662Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp1ztfdi7/_remote_module_non_scriptable.py
2022-05-18T04:02:11.2475127Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4jxjehou
2022-05-18T04:02:11.2475875Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4jxjehou/_remote_module_non_scriptable.py
2022-05-18T04:02:11.2517628Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5eucly8z
2022-05-18T04:02:11.2519376Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5eucly8z/_remote_module_non_scriptable.py
2022-05-18T04:02:11.3715193Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:02:11.4025082Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:02:11.4992037Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:02:11.5032194Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:02:12.0140330Z ok (1.828s)
2022-05-18T04:02:12.0140600Z 
2022-05-18T04:02:12.0141104Z ----------------------------------------------------------------------
2022-05-18T04:02:12.0141362Z Ran 1 test in 1.828s
2022-05-18T04:02:12.0141493Z 
2022-05-18T04:02:12.0141555Z OK
2022-05-18T04:02:12.0141647Z 
2022-05-18T04:02:12.0141730Z Generating XML reports...
2022-05-18T04:02:12.0175756Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040210.xml
2022-05-18T04:02:12.8040757Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5px76te9
2022-05-18T04:02:12.8041732Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5px76te9/_remote_module_non_scriptable.py
2022-05-18T04:02:13.0624532Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:02:13.0634205Z 
2022-05-18T04:02:13.0634316Z Running tests...
2022-05-18T04:02:13.0634907Z ----------------------------------------------------------------------
2022-05-18T04:02:13.3917097Z   test_invalid_names (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20930
2022-05-18T04:02:13.3938820Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20931
2022-05-18T04:02:13.3962443Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20932
2022-05-18T04:02:13.3987220Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 20933
2022-05-18T04:02:14.0271447Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppz70a0lt
2022-05-18T04:02:14.0272233Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppz70a0lt/_remote_module_non_scriptable.py
2022-05-18T04:02:14.0360716Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2mascg0r
2022-05-18T04:02:14.0361949Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2mascg0r/_remote_module_non_scriptable.py
2022-05-18T04:02:14.0590680Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphsr9w5bb
2022-05-18T04:02:14.0592026Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphsr9w5bb/_remote_module_non_scriptable.py
2022-05-18T04:02:14.0655324Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1c8tut3i
2022-05-18T04:02:14.0657426Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1c8tut3i/_remote_module_non_scriptable.py
2022-05-18T04:02:14.2749095Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:02:14.2833478Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:02:14.3080354Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:02:14.3153529Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:02:14.5021941Z ok (1.438s)
2022-05-18T04:02:14.5022180Z 
2022-05-18T04:02:14.5022643Z ----------------------------------------------------------------------
2022-05-18T04:02:14.5023222Z Ran 1 test in 1.439s
2022-05-18T04:02:14.5023696Z 
2022-05-18T04:02:14.5023794Z OK
2022-05-18T04:02:14.5023953Z 
2022-05-18T04:02:14.5024092Z Generating XML reports...
2022-05-18T04:02:14.5058962Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040213.xml
2022-05-18T04:02:15.2547164Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp62lma59n
2022-05-18T04:02:15.2548045Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp62lma59n/_remote_module_non_scriptable.py
2022-05-18T04:02:15.5156082Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:02:15.5165937Z 
2022-05-18T04:02:15.5166179Z Running tests...
2022-05-18T04:02:15.5166522Z ----------------------------------------------------------------------
2022-05-18T04:02:15.8426971Z   test_local_rref_no_fork (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20985
2022-05-18T04:02:15.8450172Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20986
2022-05-18T04:02:15.8473394Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20987
2022-05-18T04:02:15.8498842Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 20988
2022-05-18T04:02:16.5065107Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5_d_2uyf
2022-05-18T04:02:16.5065879Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5_d_2uyf/_remote_module_non_scriptable.py
2022-05-18T04:02:16.5074128Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6os0lgds
2022-05-18T04:02:16.5076079Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6os0lgds/_remote_module_non_scriptable.py
2022-05-18T04:02:16.5192320Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbkpxx8p5
2022-05-18T04:02:16.5193095Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbkpxx8p5/_remote_module_non_scriptable.py
2022-05-18T04:02:16.5438087Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppl9j74vd
2022-05-18T04:02:16.5438822Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppl9j74vd/_remote_module_non_scriptable.py
2022-05-18T04:02:16.7576682Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:02:16.7606049Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:02:16.7715114Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:02:16.7938333Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:02:17.2538247Z ok (1.737s)
2022-05-18T04:02:17.2538473Z 
2022-05-18T04:02:17.2539009Z ----------------------------------------------------------------------
2022-05-18T04:02:17.2539358Z Ran 1 test in 1.737s
2022-05-18T04:02:17.2539504Z 
2022-05-18T04:02:17.2539566Z OK
2022-05-18T04:02:17.2539658Z 
2022-05-18T04:02:17.2539752Z Generating XML reports...
2022-05-18T04:02:17.2574615Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040215.xml
2022-05-18T04:02:18.0713705Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpatx8gptr
2022-05-18T04:02:18.0714193Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpatx8gptr/_remote_module_non_scriptable.py
2022-05-18T04:02:18.3274003Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:02:18.3283806Z 
2022-05-18T04:02:18.3283939Z Running tests...
2022-05-18T04:02:18.3284516Z ----------------------------------------------------------------------
2022-05-18T04:02:18.6461211Z   test_local_shutdown (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21204
2022-05-18T04:02:18.6483564Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21205
2022-05-18T04:02:18.6507286Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21206
2022-05-18T04:02:18.6531164Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 21207
2022-05-18T04:02:19.2566997Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbjmhyedi
2022-05-18T04:02:19.2568334Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbjmhyedi/_remote_module_non_scriptable.py
2022-05-18T04:02:19.2771717Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg3o2pjms
2022-05-18T04:02:19.2772771Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg3o2pjms/_remote_module_non_scriptable.py
2022-05-18T04:02:19.2774130Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpriablphq
2022-05-18T04:02:19.2777388Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpriablphq/_remote_module_non_scriptable.py
2022-05-18T04:02:19.2850343Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpajpwoucr
2022-05-18T04:02:19.2851847Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpajpwoucr/_remote_module_non_scriptable.py
2022-05-18T04:02:19.5066698Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:02:19.5247368Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:02:19.5259299Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:02:19.5315042Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:02:19.7202832Z [W tensorpipe_agent.cpp:728] RPC agent for worker1 encountered error when reading incoming request from worker0: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2022-05-18T04:02:19.7203900Z [W tensorpipe_agent.cpp:728] RPC agent for worker3 encountered error when reading incoming request from worker0: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2022-05-18T04:02:19.7204962Z [W tensorpipe_agent.cpp:728] RPC agent for worker2 encountered error when reading incoming request from worker0: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2022-05-18T04:02:19.7205924Z [W tensorpipe_agent.cpp:728] RPC agent for worker0 encountered error when reading incoming request from worker2: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2022-05-18T04:02:19.7206453Z [W tensorpipe_agent.cpp:728] RPC agent for worker0 encountered error when reading incoming request from worker3: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2022-05-18T04:02:19.7206969Z [W tensorpipe_agent.cpp:728] RPC agent for worker0 encountered error when reading incoming request from worker1: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2022-05-18T04:02:19.9570660Z ok (1.628s)
2022-05-18T04:02:19.9570950Z 
2022-05-18T04:02:19.9571474Z ----------------------------------------------------------------------
2022-05-18T04:02:19.9572013Z Ran 1 test in 1.629s
2022-05-18T04:02:19.9572157Z 
2022-05-18T04:02:19.9572219Z OK
2022-05-18T04:02:19.9572303Z 
2022-05-18T04:02:19.9572398Z Generating XML reports...
2022-05-18T04:02:19.9605714Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040218.xml
2022-05-18T04:02:20.7104736Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8e70mddn
2022-05-18T04:02:20.7105476Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8e70mddn/_remote_module_non_scriptable.py
2022-05-18T04:02:20.9659012Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:02:20.9669235Z 
2022-05-18T04:02:20.9669552Z Running tests...
2022-05-18T04:02:20.9670476Z ----------------------------------------------------------------------
2022-05-18T04:02:21.2830441Z   test_local_shutdown_with_rpc (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21423
2022-05-18T04:02:21.2853155Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21424
2022-05-18T04:02:21.2876573Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21425
2022-05-18T04:02:21.2901012Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 21426
2022-05-18T04:02:21.9244368Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpljwzu1vh
2022-05-18T04:02:21.9245169Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpljwzu1vh/_remote_module_non_scriptable.py
2022-05-18T04:02:21.9360827Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd1a6y4kl
2022-05-18T04:02:21.9362059Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd1a6y4kl/_remote_module_non_scriptable.py
2022-05-18T04:02:21.9555671Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp19l5995l
2022-05-18T04:02:21.9557205Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp19l5995l/_remote_module_non_scriptable.py
2022-05-18T04:02:21.9755258Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp27ogn41g
2022-05-18T04:02:21.9756202Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp27ogn41g/_remote_module_non_scriptable.py
2022-05-18T04:02:22.1720072Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:02:22.1850216Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:02:22.2052829Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:02:22.2211656Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:02:22.4801383Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:02:22.4848195Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:02:22.4849182Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:02:22.4850450Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:02:22.4851334Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T04:02:22.4852489Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:02:22.4853652Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:02:22.4902654Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:02:22.4962269Z [W tensorpipe_agent.cpp:728] RPC agent for worker3 encountered error when reading incoming request from worker2: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2022-05-18T04:02:22.4963140Z [W tensorpipe_agent.cpp:728] RPC agent for worker2 encountered error when reading incoming request from worker1: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2022-05-18T04:02:22.4963982Z [W tensorpipe_agent.cpp:728] RPC agent for worker0 encountered error when reading incoming request from worker1: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2022-05-18T04:02:22.4964816Z [W tensorpipe_agent.cpp:728] RPC agent for worker0 encountered error when reading incoming request from worker2: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2022-05-18T04:02:22.4965645Z [W tensorpipe_agent.cpp:728] RPC agent for worker0 encountered error when reading incoming request from worker3: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2022-05-18T04:02:22.4966783Z [W tensorpipe_agent.cpp:728] RPC agent for worker1 encountered error when reading incoming request from worker0: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2022-05-18T04:02:22.4967624Z [W tensorpipe_agent.cpp:728] RPC agent for worker2 encountered error when reading incoming request from worker0: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2022-05-18T04:02:22.4968590Z [W tensorpipe_agent.cpp:728] RPC agent for worker3 encountered error when reading incoming request from worker0: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2022-05-18T04:02:22.6941309Z ok (1.727s)
2022-05-18T04:02:22.6941661Z 
2022-05-18T04:02:22.6942192Z ----------------------------------------------------------------------
2022-05-18T04:02:22.6942526Z Ran 1 test in 1.727s
2022-05-18T04:02:22.6942629Z 
2022-05-18T04:02:22.6942688Z OK
2022-05-18T04:02:22.6942779Z 
2022-05-18T04:02:22.6943082Z Generating XML reports...
2022-05-18T04:02:22.6976251Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040220.xml
2022-05-18T04:02:23.4727127Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdhcyb6e3
2022-05-18T04:02:23.4728149Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdhcyb6e3/_remote_module_non_scriptable.py
2022-05-18T04:02:23.7244916Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:02:23.7254462Z 
2022-05-18T04:02:23.7254775Z Running tests...
2022-05-18T04:02:23.7255418Z ----------------------------------------------------------------------
2022-05-18T04:02:24.0390491Z   test_local_value_not_on_owner (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21654
2022-05-18T04:02:24.0412754Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21655
2022-05-18T04:02:24.0435785Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21656
2022-05-18T04:02:24.0459847Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 21657
2022-05-18T04:02:24.6618822Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8vtof9x8
2022-05-18T04:02:24.6619742Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8vtof9x8/_remote_module_non_scriptable.py
2022-05-18T04:02:24.6725509Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg_82bmcd
2022-05-18T04:02:24.6726754Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg_82bmcd/_remote_module_non_scriptable.py
2022-05-18T04:02:24.6777352Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgo2yugfr
2022-05-18T04:02:24.6779493Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgo2yugfr/_remote_module_non_scriptable.py
2022-05-18T04:02:24.6908006Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpug32i7io
2022-05-18T04:02:24.6910423Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpug32i7io/_remote_module_non_scriptable.py
2022-05-18T04:02:24.9075006Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:02:24.9188523Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:02:24.9282718Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:02:24.9376026Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:02:25.4500062Z ok (1.724s)
2022-05-18T04:02:25.4500305Z 
2022-05-18T04:02:25.4500661Z ----------------------------------------------------------------------
2022-05-18T04:02:25.4501192Z Ran 1 test in 1.724s
2022-05-18T04:02:25.4501309Z 
2022-05-18T04:02:25.4501370Z OK
2022-05-18T04:02:25.4501460Z 
2022-05-18T04:02:25.4501543Z Generating XML reports...
2022-05-18T04:02:25.4534720Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040223.xml
2022-05-18T04:02:26.2273297Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkfptawa2
2022-05-18T04:02:26.2274105Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkfptawa2/_remote_module_non_scriptable.py
2022-05-18T04:02:26.4821324Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:02:26.4831459Z 
2022-05-18T04:02:26.4831731Z Running tests...
2022-05-18T04:02:26.4832121Z ----------------------------------------------------------------------
2022-05-18T04:02:26.8011700Z   test_mark_future_twice (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21873
2022-05-18T04:02:26.8033888Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21874
2022-05-18T04:02:26.8056986Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21875
2022-05-18T04:02:26.8080943Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 21876
2022-05-18T04:02:27.4495607Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp23zlox6a
2022-05-18T04:02:27.4496351Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp23zlox6a/_remote_module_non_scriptable.py
2022-05-18T04:02:27.4661903Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp34hjiwc4
2022-05-18T04:02:27.4663227Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp34hjiwc4/_remote_module_non_scriptable.py
2022-05-18T04:02:27.4961592Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpos9jiezs
2022-05-18T04:02:27.4962388Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpos9jiezs/_remote_module_non_scriptable.py
2022-05-18T04:02:27.5045951Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp061y_t1g
2022-05-18T04:02:27.5047180Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp061y_t1g/_remote_module_non_scriptable.py
2022-05-18T04:02:27.6998039Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:02:27.7138083Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:02:27.7436238Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:02:27.7544735Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:02:28.2123072Z ok (1.729s)
2022-05-18T04:02:28.2123317Z 
2022-05-18T04:02:28.2123820Z ----------------------------------------------------------------------
2022-05-18T04:02:28.2124097Z Ran 1 test in 1.729s
2022-05-18T04:02:28.2124231Z 
2022-05-18T04:02:28.2124292Z OK
2022-05-18T04:02:28.2124385Z 
2022-05-18T04:02:28.2124466Z Generating XML reports...
2022-05-18T04:02:28.2162220Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040226.xml
2022-05-18T04:02:28.9971112Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr1g9z40s
2022-05-18T04:02:28.9973250Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr1g9z40s/_remote_module_non_scriptable.py
2022-05-18T04:02:29.2523327Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:02:29.2532491Z 
2022-05-18T04:02:29.2532625Z Running tests...
2022-05-18T04:02:29.2533241Z ----------------------------------------------------------------------
2022-05-18T04:02:29.5654539Z   test_multi_builtin_remote_ret (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22092
2022-05-18T04:02:29.5677525Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22093
2022-05-18T04:02:29.5700868Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22094
2022-05-18T04:02:29.5725866Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 22095
2022-05-18T04:02:30.1692553Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp97rn2g2t
2022-05-18T04:02:30.1693388Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp97rn2g2t/_remote_module_non_scriptable.py
2022-05-18T04:02:30.2017838Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6gk8hv5s
2022-05-18T04:02:30.2019484Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6gk8hv5s/_remote_module_non_scriptable.py
2022-05-18T04:02:30.2119998Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_vv0bb8w
2022-05-18T04:02:30.2122249Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_vv0bb8w/_remote_module_non_scriptable.py
2022-05-18T04:02:30.2246061Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsmp172l6
2022-05-18T04:02:30.2247706Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsmp172l6/_remote_module_non_scriptable.py
2022-05-18T04:02:30.4162038Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:02:30.4523405Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:02:30.4579241Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:02:30.4734719Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:02:30.9766423Z ok (1.723s)
2022-05-18T04:02:30.9766648Z 
2022-05-18T04:02:30.9767111Z ----------------------------------------------------------------------
2022-05-18T04:02:30.9801094Z Ran 1 test in 1.723s
2022-05-18T04:02:30.9801320Z 
2022-05-18T04:02:30.9801429Z OK
2022-05-18T04:02:30.9801586Z 
2022-05-18T04:02:30.9801741Z Generating XML reports...
2022-05-18T04:02:30.9802639Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040229.xml
2022-05-18T04:02:31.7509806Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_jqp2p2c
2022-05-18T04:02:31.7510525Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_jqp2p2c/_remote_module_non_scriptable.py
2022-05-18T04:02:32.0092988Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:02:32.0101582Z 
2022-05-18T04:02:32.0101686Z Running tests...
2022-05-18T04:02:32.0102334Z ----------------------------------------------------------------------
2022-05-18T04:02:32.3296772Z   test_multi_layer_nested_async_rpc (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22311
2022-05-18T04:02:32.3319933Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22312
2022-05-18T04:02:32.3343525Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22313
2022-05-18T04:02:32.3367233Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 22314
2022-05-18T04:02:32.9515939Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzefop70a
2022-05-18T04:02:32.9516782Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzefop70a/_remote_module_non_scriptable.py
2022-05-18T04:02:32.9581059Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcgxogurq
2022-05-18T04:02:32.9582688Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcgxogurq/_remote_module_non_scriptable.py
2022-05-18T04:02:32.9785876Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnj1uzrc3
2022-05-18T04:02:32.9786607Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnj1uzrc3/_remote_module_non_scriptable.py
2022-05-18T04:02:32.9789309Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp54lzbcbv
2022-05-18T04:02:32.9792172Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp54lzbcbv/_remote_module_non_scriptable.py
2022-05-18T04:02:33.1994138Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:02:33.2031733Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:02:33.2278808Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:02:33.2283466Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:02:33.7406946Z ok (1.730s)
2022-05-18T04:02:33.7407172Z 
2022-05-18T04:02:33.7407625Z ----------------------------------------------------------------------
2022-05-18T04:02:33.7408027Z Ran 1 test in 1.730s
2022-05-18T04:02:33.7408205Z 
2022-05-18T04:02:33.7408304Z OK
2022-05-18T04:02:33.7408445Z 
2022-05-18T04:02:33.7408592Z Generating XML reports...
2022-05-18T04:02:33.7443164Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040232.xml
2022-05-18T04:02:34.5233368Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpousayk9v
2022-05-18T04:02:34.5234079Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpousayk9v/_remote_module_non_scriptable.py
2022-05-18T04:02:34.7796990Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:02:34.7807367Z 
2022-05-18T04:02:34.7807655Z Running tests...
2022-05-18T04:02:34.7808231Z ----------------------------------------------------------------------
2022-05-18T04:02:35.1086381Z   test_multi_py_udf_remote (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22530
2022-05-18T04:02:35.1109319Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22531
2022-05-18T04:02:35.1132744Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22532
2022-05-18T04:02:35.1157771Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 22533
2022-05-18T04:02:35.7620236Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2b0aqrn2
2022-05-18T04:02:35.7621046Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2b0aqrn2/_remote_module_non_scriptable.py
2022-05-18T04:02:35.7633732Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeg82anp0
2022-05-18T04:02:35.7646215Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeg82anp0/_remote_module_non_scriptable.py
2022-05-18T04:02:35.7957709Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpaj_8e3at
2022-05-18T04:02:35.7959045Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpaj_8e3at/_remote_module_non_scriptable.py
2022-05-18T04:02:35.8056665Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsj773ptn
2022-05-18T04:02:35.8057755Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsj773ptn/_remote_module_non_scriptable.py
2022-05-18T04:02:36.0137407Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:02:36.0179275Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:02:36.0476461Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:02:36.0581855Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:02:36.5197936Z ok (1.739s)
2022-05-18T04:02:36.5198343Z 
2022-05-18T04:02:36.5198866Z ----------------------------------------------------------------------
2022-05-18T04:02:36.5199351Z Ran 1 test in 1.739s
2022-05-18T04:02:36.5199775Z 
2022-05-18T04:02:36.5199824Z OK
2022-05-18T04:02:36.5199916Z 
2022-05-18T04:02:36.5200011Z Generating XML reports...
2022-05-18T04:02:36.5233731Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040234.xml
2022-05-18T04:02:37.3095656Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpopkb07yg
2022-05-18T04:02:37.3096375Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpopkb07yg/_remote_module_non_scriptable.py
2022-05-18T04:02:37.5640133Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:02:37.5649812Z 
2022-05-18T04:02:37.5649911Z Running tests...
2022-05-18T04:02:37.5650874Z ----------------------------------------------------------------------
2022-05-18T04:02:37.8805870Z   test_multi_rpc (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22749
2022-05-18T04:02:37.8828626Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22750
2022-05-18T04:02:37.8852445Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22751
2022-05-18T04:02:37.8877096Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 22752
2022-05-18T04:02:38.5422659Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp22w9hbo8
2022-05-18T04:02:38.5423591Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp22w9hbo8/_remote_module_non_scriptable.py
2022-05-18T04:02:38.5428328Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu2iufysl
2022-05-18T04:02:38.5430609Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu2iufysl/_remote_module_non_scriptable.py
2022-05-18T04:02:38.5451065Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz82gkh5a
2022-05-18T04:02:38.5453196Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz82gkh5a/_remote_module_non_scriptable.py
2022-05-18T04:02:38.5499634Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbs_5dd9n
2022-05-18T04:02:38.5502138Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbs_5dd9n/_remote_module_non_scriptable.py
2022-05-18T04:02:38.7905710Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:02:38.7934351Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:02:38.7939254Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:02:38.8017263Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:02:39.2918274Z ok (1.726s)
2022-05-18T04:02:39.2918552Z 
2022-05-18T04:02:39.2918995Z ----------------------------------------------------------------------
2022-05-18T04:02:39.2919252Z Ran 1 test in 1.727s
2022-05-18T04:02:39.2919398Z 
2022-05-18T04:02:39.2919462Z OK
2022-05-18T04:02:39.2919542Z 
2022-05-18T04:02:39.2919635Z Generating XML reports...
2022-05-18T04:02:39.2953626Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040237.xml
2022-05-18T04:02:40.0785378Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph078n9zi
2022-05-18T04:02:40.0785844Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph078n9zi/_remote_module_non_scriptable.py
2022-05-18T04:02:40.3317561Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:02:40.3327130Z 
2022-05-18T04:02:40.3327234Z Running tests...
2022-05-18T04:02:40.3327640Z ----------------------------------------------------------------------
2022-05-18T04:02:40.6514881Z   test_my_parameter_server (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22968
2022-05-18T04:02:40.6538235Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22969
2022-05-18T04:02:40.6561699Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22970
2022-05-18T04:02:40.6586226Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 22971
2022-05-18T04:02:41.3096548Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8j_xia37
2022-05-18T04:02:41.3097680Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8j_xia37/_remote_module_non_scriptable.py
2022-05-18T04:02:41.3469408Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5anjxqhp
2022-05-18T04:02:41.3470234Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5anjxqhp/_remote_module_non_scriptable.py
2022-05-18T04:02:41.3776209Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5czxyt96
2022-05-18T04:02:41.3776980Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5czxyt96/_remote_module_non_scriptable.py
2022-05-18T04:02:41.3869310Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppsw14_im
2022-05-18T04:02:41.3870924Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppsw14_im/_remote_module_non_scriptable.py
2022-05-18T04:02:41.5575930Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:02:41.5970460Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:02:41.6252366Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:02:41.6356946Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:02:42.2630922Z ok (1.930s)
2022-05-18T04:02:42.2631195Z 
2022-05-18T04:02:42.2631706Z ----------------------------------------------------------------------
2022-05-18T04:02:42.2631974Z Ran 1 test in 1.930s
2022-05-18T04:02:42.2632092Z 
2022-05-18T04:02:42.2632154Z OK
2022-05-18T04:02:42.2632247Z 
2022-05-18T04:02:42.2632343Z Generating XML reports...
2022-05-18T04:02:42.2665843Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040240.xml
2022-05-18T04:02:43.0464465Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp43wsmi09
2022-05-18T04:02:43.0464974Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp43wsmi09/_remote_module_non_scriptable.py
2022-05-18T04:02:43.2996340Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:02:43.3005587Z 
2022-05-18T04:02:43.3005809Z Running tests...
2022-05-18T04:02:43.3006418Z ----------------------------------------------------------------------
2022-05-18T04:02:43.6169879Z   test_nested_remote (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23223
2022-05-18T04:02:43.6192751Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23224
2022-05-18T04:02:43.6215777Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23225
2022-05-18T04:02:43.6239731Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 23226
2022-05-18T04:02:44.2369399Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxmgorgc1
2022-05-18T04:02:44.2370170Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxmgorgc1/_remote_module_non_scriptable.py
2022-05-18T04:02:44.2467127Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprkzykxwq
2022-05-18T04:02:44.2468493Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprkzykxwq/_remote_module_non_scriptable.py
2022-05-18T04:02:44.2748059Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv1wm_1_v
2022-05-18T04:02:44.2748756Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgl2ne3a1
2022-05-18T04:02:44.2749570Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv1wm_1_v/_remote_module_non_scriptable.py
2022-05-18T04:02:44.2750714Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgl2ne3a1/_remote_module_non_scriptable.py
2022-05-18T04:02:44.4859888Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:02:44.4930197Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:02:44.5226323Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:02:44.5240649Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:02:45.0281160Z ok (1.727s)
2022-05-18T04:02:45.0281435Z 
2022-05-18T04:02:45.0281907Z ----------------------------------------------------------------------
2022-05-18T04:02:45.0282175Z Ran 1 test in 1.727s
2022-05-18T04:02:45.0282307Z 
2022-05-18T04:02:45.0282372Z OK
2022-05-18T04:02:45.0282451Z 
2022-05-18T04:02:45.0282544Z Generating XML reports...
2022-05-18T04:02:45.0317082Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040243.xml
2022-05-18T04:02:45.8156622Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5_90c51e
2022-05-18T04:02:45.8157419Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5_90c51e/_remote_module_non_scriptable.py
2022-05-18T04:02:46.0699780Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:02:46.0709647Z 
2022-05-18T04:02:46.0709823Z Running tests...
2022-05-18T04:02:46.0710218Z ----------------------------------------------------------------------
2022-05-18T04:02:46.3867855Z   test_nested_rpc (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23442
2022-05-18T04:02:46.3889913Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23443
2022-05-18T04:02:46.3913474Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23444
2022-05-18T04:02:46.3937761Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 23445
2022-05-18T04:02:46.9666682Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp85jzeodd
2022-05-18T04:02:46.9667302Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjot45dvo
2022-05-18T04:02:46.9667949Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp85jzeodd/_remote_module_non_scriptable.py
2022-05-18T04:02:46.9668635Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjot45dvo/_remote_module_non_scriptable.py
2022-05-18T04:02:46.9796392Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0yx8gny_
2022-05-18T04:02:46.9797601Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0yx8gny_/_remote_module_non_scriptable.py
2022-05-18T04:02:47.0229723Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppx6pgbsx
2022-05-18T04:02:47.0230573Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppx6pgbsx/_remote_module_non_scriptable.py
2022-05-18T04:02:47.2180941Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:02:47.2186167Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:02:47.2318009Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:02:47.2775087Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:02:47.7978399Z ok (1.726s)
2022-05-18T04:02:47.7978601Z 
2022-05-18T04:02:47.7979002Z ----------------------------------------------------------------------
2022-05-18T04:02:47.7979324Z Ran 1 test in 1.727s
2022-05-18T04:02:47.7979455Z 
2022-05-18T04:02:47.7979843Z OK
2022-05-18T04:02:47.7979966Z 
2022-05-18T04:02:47.7980074Z Generating XML reports...
2022-05-18T04:02:47.8014619Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040246.xml
2022-05-18T04:02:48.6356932Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk85wsaes
2022-05-18T04:02:48.6357731Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk85wsaes/_remote_module_non_scriptable.py
2022-05-18T04:02:48.8904483Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:02:48.8914417Z 
2022-05-18T04:02:48.8914504Z Running tests...
2022-05-18T04:02:48.8915626Z ----------------------------------------------------------------------
2022-05-18T04:02:49.2215974Z   test_nested_rref (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23661
2022-05-18T04:02:49.2239451Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23662
2022-05-18T04:02:49.2263063Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23663
2022-05-18T04:02:49.2287323Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 23664
2022-05-18T04:02:49.8252924Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo6lrbwhj
2022-05-18T04:02:49.8254138Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo6lrbwhj/_remote_module_non_scriptable.py
2022-05-18T04:02:49.8338677Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprin7ovty
2022-05-18T04:02:49.8339578Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprin7ovty/_remote_module_non_scriptable.py
2022-05-18T04:02:49.8693066Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0gka_92i
2022-05-18T04:02:49.8694090Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0gka_92i/_remote_module_non_scriptable.py
2022-05-18T04:02:49.8734227Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuy84e0i4
2022-05-18T04:02:49.8735297Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuy84e0i4/_remote_module_non_scriptable.py
2022-05-18T04:02:50.0763612Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:02:50.0844376Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:02:50.1193141Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:02:50.1247947Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:02:50.6327354Z ok (1.741s)
2022-05-18T04:02:50.6327622Z 
2022-05-18T04:02:50.6328118Z ----------------------------------------------------------------------
2022-05-18T04:02:50.6328604Z Ran 1 test in 1.741s
2022-05-18T04:02:50.6328748Z 
2022-05-18T04:02:50.6328835Z OK
2022-05-18T04:02:50.6328927Z 
2022-05-18T04:02:50.6329023Z Generating XML reports...
2022-05-18T04:02:50.6362596Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040248.xml
2022-05-18T04:02:51.4209685Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7b5yhtyw
2022-05-18T04:02:51.4210222Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7b5yhtyw/_remote_module_non_scriptable.py
2022-05-18T04:02:51.6748482Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:02:51.6757195Z 
2022-05-18T04:02:51.6757331Z Running tests...
2022-05-18T04:02:51.6757932Z ----------------------------------------------------------------------
2022-05-18T04:02:51.9917005Z   test_nested_rref_stress (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23880
2022-05-18T04:02:51.9941295Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23881
2022-05-18T04:02:51.9964633Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23882
2022-05-18T04:02:51.9989229Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 23883
2022-05-18T04:02:52.6175845Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7hsz2sp8
2022-05-18T04:02:52.6176537Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7hsz2sp8/_remote_module_non_scriptable.py
2022-05-18T04:02:52.6254944Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpar7r7jiw
2022-05-18T04:02:52.6255875Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpar7r7jiw/_remote_module_non_scriptable.py
2022-05-18T04:02:52.6444680Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj8tzu22i
2022-05-18T04:02:52.6445387Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj8tzu22i/_remote_module_non_scriptable.py
2022-05-18T04:02:52.6622625Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6oaq39n4
2022-05-18T04:02:52.6624563Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6oaq39n4/_remote_module_non_scriptable.py
2022-05-18T04:02:52.8764159Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:02:52.8822046Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:02:52.9012807Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:02:52.9170939Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:02:53.5032769Z ok (1.827s)
2022-05-18T04:02:53.5033069Z 
2022-05-18T04:02:53.5033547Z ----------------------------------------------------------------------
2022-05-18T04:02:53.5033813Z Ran 1 test in 1.827s
2022-05-18T04:02:53.5033931Z 
2022-05-18T04:02:53.5034012Z OK
2022-05-18T04:02:53.5034105Z 
2022-05-18T04:02:53.5034201Z Generating XML reports...
2022-05-18T04:02:53.5068314Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040251.xml
2022-05-18T04:02:54.2886298Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8e93bvbn
2022-05-18T04:02:54.2887086Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8e93bvbn/_remote_module_non_scriptable.py
2022-05-18T04:02:54.5419711Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:02:54.5429414Z 
2022-05-18T04:02:54.5429511Z Running tests...
2022-05-18T04:02:54.5430601Z ----------------------------------------------------------------------
2022-05-18T04:02:54.8603998Z   test_non_cont_tensors (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24099
2022-05-18T04:02:54.8627199Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24100
2022-05-18T04:02:54.8650123Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24101
2022-05-18T04:02:54.8674832Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 24102
2022-05-18T04:02:55.5511553Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqqg2qfpb
2022-05-18T04:02:55.5512321Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqqg2qfpb/_remote_module_non_scriptable.py
2022-05-18T04:02:55.5779404Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3jmc230f
2022-05-18T04:02:55.5780139Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3jmc230f/_remote_module_non_scriptable.py
2022-05-18T04:02:55.5817874Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplh0cq1jp
2022-05-18T04:02:55.5819230Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplh0cq1jp/_remote_module_non_scriptable.py
2022-05-18T04:02:55.6231309Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5u0qnqu3
2022-05-18T04:02:55.6232286Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5u0qnqu3/_remote_module_non_scriptable.py
2022-05-18T04:02:55.7999699Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:02:55.8280502Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:02:55.8295696Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:02:55.8682229Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:02:56.2715226Z ok (1.728s)
2022-05-18T04:02:56.2715501Z 
2022-05-18T04:02:56.2716002Z ----------------------------------------------------------------------
2022-05-18T04:02:56.2716254Z Ran 1 test in 1.728s
2022-05-18T04:02:56.2716356Z 
2022-05-18T04:02:56.2716432Z OK
2022-05-18T04:02:56.2716524Z 
2022-05-18T04:02:56.2716618Z Generating XML reports...
2022-05-18T04:02:56.2749805Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040254.xml
2022-05-18T04:02:57.0406890Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjw3_1we4
2022-05-18T04:02:57.0407785Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjw3_1we4/_remote_module_non_scriptable.py
2022-05-18T04:02:57.2946117Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:02:57.2956102Z 
2022-05-18T04:02:57.2956239Z Running tests...
2022-05-18T04:02:57.2956793Z ----------------------------------------------------------------------
2022-05-18T04:02:57.6116632Z   test_non_garbage_collected_user_rref_due_to_local_circular_dependency (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24318
2022-05-18T04:02:57.6139887Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24319
2022-05-18T04:02:57.6163751Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24320
2022-05-18T04:02:57.6187593Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 24321
2022-05-18T04:02:58.2852213Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpana4fwqy
2022-05-18T04:02:58.2871195Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpana4fwqy/_remote_module_non_scriptable.py
2022-05-18T04:02:58.3261751Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpii1xpg59
2022-05-18T04:02:58.3262536Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpii1xpg59/_remote_module_non_scriptable.py
2022-05-18T04:02:58.3471676Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_0q3c905
2022-05-18T04:02:58.3472428Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_0q3c905/_remote_module_non_scriptable.py
2022-05-18T04:02:58.3738354Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsdt__sxo
2022-05-18T04:02:58.3739367Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsdt__sxo/_remote_module_non_scriptable.py
2022-05-18T04:02:58.5336780Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:02:58.5769056Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:02:58.5932616Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:02:58.6211007Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:02:59.0229225Z ok (1.727s)
2022-05-18T04:02:59.0229513Z 
2022-05-18T04:02:59.0229949Z ----------------------------------------------------------------------
2022-05-18T04:02:59.0230203Z Ran 1 test in 1.727s
2022-05-18T04:02:59.0230556Z 
2022-05-18T04:02:59.0230604Z OK
2022-05-18T04:02:59.0230697Z 
2022-05-18T04:02:59.0230792Z Generating XML reports...
2022-05-18T04:02:59.0263678Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040257.xml
2022-05-18T04:02:59.8028127Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd7rl0hvt
2022-05-18T04:02:59.8028975Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd7rl0hvt/_remote_module_non_scriptable.py
2022-05-18T04:03:00.0554646Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:03:00.0563958Z 
2022-05-18T04:03:00.0564136Z Running tests...
2022-05-18T04:03:00.0564559Z ----------------------------------------------------------------------
2022-05-18T04:03:00.3678314Z   test_nonzero (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24537
2022-05-18T04:03:00.3701111Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24538
2022-05-18T04:03:00.3724491Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24539
2022-05-18T04:03:00.3748209Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 24540
2022-05-18T04:03:01.0740685Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfrs0clh8
2022-05-18T04:03:01.0741419Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfrs0clh8/_remote_module_non_scriptable.py
2022-05-18T04:03:01.0869545Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg94vj6f9
2022-05-18T04:03:01.0871791Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg94vj6f9/_remote_module_non_scriptable.py
2022-05-18T04:03:01.1087154Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl_sb_vwe
2022-05-18T04:03:01.1088297Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl_sb_vwe/_remote_module_non_scriptable.py
2022-05-18T04:03:01.1265749Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5jlvq_y1
2022-05-18T04:03:01.1266623Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5jlvq_y1/_remote_module_non_scriptable.py
2022-05-18T04:03:01.3248898Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:03:01.3391876Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:03:01.3595318Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:03:01.3754888Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:03:01.7788832Z ok (1.722s)
2022-05-18T04:03:01.7789071Z 
2022-05-18T04:03:01.7789595Z ----------------------------------------------------------------------
2022-05-18T04:03:01.7790045Z Ran 1 test in 1.722s
2022-05-18T04:03:01.7790941Z 
2022-05-18T04:03:01.7791464Z OK
2022-05-18T04:03:01.7791705Z 
2022-05-18T04:03:01.7791847Z Generating XML reports...
2022-05-18T04:03:01.7824103Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040300.xml
2022-05-18T04:03:02.5627690Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpom3iepyy
2022-05-18T04:03:02.5628442Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpom3iepyy/_remote_module_non_scriptable.py
2022-05-18T04:03:02.8165367Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:03:02.8175210Z 
2022-05-18T04:03:02.8175450Z Running tests...
2022-05-18T04:03:02.8176103Z ----------------------------------------------------------------------
2022-05-18T04:03:03.1373091Z   test_owner_equality (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24756
2022-05-18T04:03:03.1395355Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24757
2022-05-18T04:03:03.1419122Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24758
2022-05-18T04:03:03.1443429Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 24759
2022-05-18T04:03:03.7717008Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl619zihq
2022-05-18T04:03:03.7717751Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl619zihq/_remote_module_non_scriptable.py
2022-05-18T04:03:03.8281101Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcgz0p3b1
2022-05-18T04:03:03.8281918Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcgz0p3b1/_remote_module_non_scriptable.py
2022-05-18T04:03:03.8284113Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdib0k6a0
2022-05-18T04:03:03.8286472Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdib0k6a0/_remote_module_non_scriptable.py
2022-05-18T04:03:03.8450360Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzzb0623d
2022-05-18T04:03:03.8451165Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzzb0623d/_remote_module_non_scriptable.py
2022-05-18T04:03:04.0216175Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:03:04.0769162Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:03:04.0777113Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:03:04.0917638Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:03:04.5482847Z ok (1.730s)
2022-05-18T04:03:04.5483106Z 
2022-05-18T04:03:04.5483659Z ----------------------------------------------------------------------
2022-05-18T04:03:04.5483935Z Ran 1 test in 1.731s
2022-05-18T04:03:04.5484052Z 
2022-05-18T04:03:04.5484114Z OK
2022-05-18T04:03:04.5484206Z 
2022-05-18T04:03:04.5484300Z Generating XML reports...
2022-05-18T04:03:04.5517755Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040302.xml
2022-05-18T04:03:05.3286480Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4gnki7o1
2022-05-18T04:03:05.3287740Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4gnki7o1/_remote_module_non_scriptable.py
2022-05-18T04:03:05.5858646Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:03:05.5869454Z 
2022-05-18T04:03:05.5869954Z Running tests...
2022-05-18T04:03:05.5870549Z ----------------------------------------------------------------------
2022-05-18T04:03:05.9067177Z   test_owner_rref_backward (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24975
2022-05-18T04:03:05.9090929Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24976
2022-05-18T04:03:05.9114597Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24977
2022-05-18T04:03:05.9139823Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 24978
2022-05-18T04:03:06.5423620Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5h9tj4jr
2022-05-18T04:03:06.5424391Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5h9tj4jr/_remote_module_non_scriptable.py
2022-05-18T04:03:06.5545059Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3r318r_9
2022-05-18T04:03:06.5545989Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3r318r_9/_remote_module_non_scriptable.py
2022-05-18T04:03:06.5597898Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl1a166_u
2022-05-18T04:03:06.5599731Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl1a166_u/_remote_module_non_scriptable.py
2022-05-18T04:03:06.5636826Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuc9wz90v
2022-05-18T04:03:06.5638767Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuc9wz90v/_remote_module_non_scriptable.py
2022-05-18T04:03:06.7893210Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:03:06.8022352Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:03:06.8060693Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:03:06.8119988Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:03:07.3180037Z ok (1.731s)
2022-05-18T04:03:07.3180379Z 
2022-05-18T04:03:07.3180919Z ----------------------------------------------------------------------
2022-05-18T04:03:07.3181184Z Ran 1 test in 1.731s
2022-05-18T04:03:07.3181298Z 
2022-05-18T04:03:07.3181388Z OK
2022-05-18T04:03:07.3181481Z 
2022-05-18T04:03:07.3181562Z Generating XML reports...
2022-05-18T04:03:07.3215037Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040305.xml
2022-05-18T04:03:08.0951656Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdbh_q0en
2022-05-18T04:03:08.0952122Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdbh_q0en/_remote_module_non_scriptable.py
2022-05-18T04:03:08.3511407Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:03:08.3521019Z 
2022-05-18T04:03:08.3521304Z Running tests...
2022-05-18T04:03:08.3521986Z ----------------------------------------------------------------------
2022-05-18T04:03:08.6699376Z   test_pass_local_rrefs (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25214
2022-05-18T04:03:08.6722466Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25215
2022-05-18T04:03:08.6745570Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25216
2022-05-18T04:03:08.6770368Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 25217
2022-05-18T04:03:09.2887583Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3fv7e1vg
2022-05-18T04:03:09.2888373Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3fv7e1vg/_remote_module_non_scriptable.py
2022-05-18T04:03:09.2898609Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgbz5ua1a
2022-05-18T04:03:09.2900687Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgbz5ua1a/_remote_module_non_scriptable.py
2022-05-18T04:03:09.2957974Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0p3o1867
2022-05-18T04:03:09.2959895Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0p3o1867/_remote_module_non_scriptable.py
2022-05-18T04:03:09.3023968Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpawf7vo2r
2022-05-18T04:03:09.3025622Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpawf7vo2r/_remote_module_non_scriptable.py
2022-05-18T04:03:09.5387383Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:03:09.5410844Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:03:09.5474326Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:03:09.5518199Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:03:10.0810635Z ok (1.729s)
2022-05-18T04:03:10.0810845Z 
2022-05-18T04:03:10.0811216Z ----------------------------------------------------------------------
2022-05-18T04:03:10.0811589Z Ran 1 test in 1.729s
2022-05-18T04:03:10.0811708Z 
2022-05-18T04:03:10.0811975Z OK
2022-05-18T04:03:10.0812069Z 
2022-05-18T04:03:10.0812176Z Generating XML reports...
2022-05-18T04:03:10.0846205Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040308.xml
2022-05-18T04:03:10.8574378Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxi8rosbb
2022-05-18T04:03:10.8575213Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxi8rosbb/_remote_module_non_scriptable.py
2022-05-18T04:03:11.1146639Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:03:11.1156060Z 
2022-05-18T04:03:11.1156151Z Running tests...
2022-05-18T04:03:11.1157148Z ----------------------------------------------------------------------
2022-05-18T04:03:11.4333080Z   test_pg_init_no_rpc_init (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25433
2022-05-18T04:03:11.4356050Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25434
2022-05-18T04:03:11.4378990Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25435
2022-05-18T04:03:11.4404238Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 25436
2022-05-18T04:03:12.0430797Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxp5jp3z_
2022-05-18T04:03:12.0432209Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxp5jp3z_/_remote_module_non_scriptable.py
2022-05-18T04:03:12.0537607Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwm8gcfiq
2022-05-18T04:03:12.0538913Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwm8gcfiq/_remote_module_non_scriptable.py
2022-05-18T04:03:12.0637331Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk1gqpx0t
2022-05-18T04:03:12.0638219Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk1gqpx0t/_remote_module_non_scriptable.py
2022-05-18T04:03:12.0642413Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3t6_l5ns
2022-05-18T04:03:12.0645015Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3t6_l5ns/_remote_module_non_scriptable.py
2022-05-18T04:03:12.2905392Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:03:12.3011491Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:03:12.3134330Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:03:12.3138675Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:03:12.3525883Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:03:12.3626822Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:03:12.3627469Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T04:03:12.3628078Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:03:12.3629015Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:03:12.3631275Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:03:12.3632168Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:03:12.3633018Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:03:12.5437947Z ok (1.428s)
2022-05-18T04:03:12.5438322Z 
2022-05-18T04:03:12.5438859Z ----------------------------------------------------------------------
2022-05-18T04:03:12.5439330Z Ran 1 test in 1.428s
2022-05-18T04:03:12.5439449Z 
2022-05-18T04:03:12.5439519Z OK
2022-05-18T04:03:12.5439612Z 
2022-05-18T04:03:12.5439710Z Generating XML reports...
2022-05-18T04:03:12.5473932Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040311.xml
2022-05-18T04:03:13.3047373Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptqw2qtvs
2022-05-18T04:03:13.3047886Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptqw2qtvs/_remote_module_non_scriptable.py
2022-05-18T04:03:13.5588699Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:03:13.5598464Z 
2022-05-18T04:03:13.5598802Z Running tests...
2022-05-18T04:03:13.5599233Z ----------------------------------------------------------------------
2022-05-18T04:03:13.8729467Z   test_pickle_future (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25500
2022-05-18T04:03:13.8751429Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25501
2022-05-18T04:03:13.8774584Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25502
2022-05-18T04:03:13.8798543Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 25503
2022-05-18T04:03:14.4844389Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1uaw5daq
2022-05-18T04:03:14.4845526Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1uaw5daq/_remote_module_non_scriptable.py
2022-05-18T04:03:14.5310277Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphnb9glhv
2022-05-18T04:03:14.5311873Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphnb9glhv/_remote_module_non_scriptable.py
2022-05-18T04:03:14.5872299Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpat380uul
2022-05-18T04:03:14.5873731Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpat380uul/_remote_module_non_scriptable.py
2022-05-18T04:03:14.6301242Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyf8vhbn0
2022-05-18T04:03:14.6302013Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyf8vhbn0/_remote_module_non_scriptable.py
2022-05-18T04:03:14.7329349Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:03:14.7790886Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:03:14.8486764Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:03:14.8762087Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:03:15.3840105Z ok (1.824s)
2022-05-18T04:03:15.3840321Z 
2022-05-18T04:03:15.3840832Z ----------------------------------------------------------------------
2022-05-18T04:03:15.3841240Z Ran 1 test in 1.824s
2022-05-18T04:03:15.3841356Z 
2022-05-18T04:03:15.3841404Z OK
2022-05-18T04:03:15.3841494Z 
2022-05-18T04:03:15.3841585Z Generating XML reports...
2022-05-18T04:03:15.3875142Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040313.xml
2022-05-18T04:03:16.1646498Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4pfkax9q
2022-05-18T04:03:16.1647397Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4pfkax9q/_remote_module_non_scriptable.py
2022-05-18T04:03:16.4177695Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:03:16.4187340Z 
2022-05-18T04:03:16.4187467Z Running tests...
2022-05-18T04:03:16.4188025Z ----------------------------------------------------------------------
2022-05-18T04:03:16.7354087Z   test_profiler_export_trace (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25719
2022-05-18T04:03:16.7376948Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25720
2022-05-18T04:03:16.7400157Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25721
2022-05-18T04:03:16.7424628Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 25722
2022-05-18T04:03:17.4356874Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpothb_2vh
2022-05-18T04:03:17.4357621Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpothb_2vh/_remote_module_non_scriptable.py
2022-05-18T04:03:17.4601299Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt1e99h28
2022-05-18T04:03:17.4602455Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt1e99h28/_remote_module_non_scriptable.py
2022-05-18T04:03:17.4679302Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxnf4h0y3
2022-05-18T04:03:17.4680810Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxnf4h0y3/_remote_module_non_scriptable.py
2022-05-18T04:03:17.4847814Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplmyscp82
2022-05-18T04:03:17.4849212Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplmyscp82/_remote_module_non_scriptable.py
2022-05-18T04:03:17.6831717Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:03:17.7100459Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:03:17.7181183Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:03:17.7293634Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:03:18.1464535Z ok (1.727s)
2022-05-18T04:03:18.1465809Z 
2022-05-18T04:03:18.1466126Z ----------------------------------------------------------------------
2022-05-18T04:03:18.1466396Z Ran 1 test in 1.728s
2022-05-18T04:03:18.1466510Z 
2022-05-18T04:03:18.1466571Z OK
2022-05-18T04:03:18.1466648Z 
2022-05-18T04:03:18.1466744Z Generating XML reports...
2022-05-18T04:03:18.1499669Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040316.xml
2022-05-18T04:03:18.9178874Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpya45t8sh
2022-05-18T04:03:18.9179575Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpya45t8sh/_remote_module_non_scriptable.py
2022-05-18T04:03:19.1710158Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:03:19.1719455Z 
2022-05-18T04:03:19.1719747Z Running tests...
2022-05-18T04:03:19.1720420Z ----------------------------------------------------------------------
2022-05-18T04:03:19.4850786Z   test_profiler_remote_events_profiled (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25938
2022-05-18T04:03:19.4873902Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25939
2022-05-18T04:03:19.4897196Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25940
2022-05-18T04:03:19.4920642Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 25941
2022-05-18T04:03:20.0758042Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplc1cc5xo
2022-05-18T04:03:20.0758821Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplc1cc5xo/_remote_module_non_scriptable.py
2022-05-18T04:03:20.1214306Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkcjkpgag
2022-05-18T04:03:20.1215129Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkcjkpgag/_remote_module_non_scriptable.py
2022-05-18T04:03:20.1216304Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnmlb41qs
2022-05-18T04:03:20.1219589Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnmlb41qs/_remote_module_non_scriptable.py
2022-05-18T04:03:20.1382427Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6np_qxv9
2022-05-18T04:03:20.1383631Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6np_qxv9/_remote_module_non_scriptable.py
2022-05-18T04:03:20.3220919Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:03:20.3695412Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:03:20.3706877Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:03:20.3843001Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:03:20.7959166Z ok (1.624s)
2022-05-18T04:03:20.7959375Z 
2022-05-18T04:03:20.7959882Z ----------------------------------------------------------------------
2022-05-18T04:03:20.7960338Z Ran 1 test in 1.624s
2022-05-18T04:03:20.7960456Z 
2022-05-18T04:03:20.7960509Z OK
2022-05-18T04:03:20.7960600Z 
2022-05-18T04:03:20.7960696Z Generating XML reports...
2022-05-18T04:03:20.7994679Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040319.xml
2022-05-18T04:03:21.5712183Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9i99107v
2022-05-18T04:03:21.5712689Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9i99107v/_remote_module_non_scriptable.py
2022-05-18T04:03:21.8227059Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:03:21.8236937Z 
2022-05-18T04:03:21.8237259Z Running tests...
2022-05-18T04:03:21.8237906Z ----------------------------------------------------------------------
2022-05-18T04:03:22.1369856Z   test_profiler_remote_events_profiled_single_threaded (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26157
2022-05-18T04:03:22.1393182Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26158
2022-05-18T04:03:22.1416704Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26159
2022-05-18T04:03:22.1440917Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 26160
2022-05-18T04:03:22.7720231Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsy72h082
2022-05-18T04:03:22.7721242Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsy72h082/_remote_module_non_scriptable.py
2022-05-18T04:03:22.7865343Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3v2bg645
2022-05-18T04:03:22.7866370Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3v2bg645/_remote_module_non_scriptable.py
2022-05-18T04:03:22.7896827Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd9msm7mb
2022-05-18T04:03:22.7898868Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd9msm7mb/_remote_module_non_scriptable.py
2022-05-18T04:03:22.8092365Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0rszj5jm
2022-05-18T04:03:22.8093113Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0rszj5jm/_remote_module_non_scriptable.py
2022-05-18T04:03:23.0226714Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:03:23.0336399Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:03:23.0369878Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:03:23.0553824Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:03:23.5480957Z ok (1.724s)
2022-05-18T04:03:23.5481156Z 
2022-05-18T04:03:23.5481652Z ----------------------------------------------------------------------
2022-05-18T04:03:23.5482213Z Ran 1 test in 1.724s
2022-05-18T04:03:23.5482329Z 
2022-05-18T04:03:23.5482391Z OK
2022-05-18T04:03:23.5482483Z 
2022-05-18T04:03:23.5482575Z Generating XML reports...
2022-05-18T04:03:23.5516606Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040321.xml
2022-05-18T04:03:24.3281262Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyxn8uv0y
2022-05-18T04:03:24.3282901Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyxn8uv0y/_remote_module_non_scriptable.py
2022-05-18T04:03:24.5808997Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:03:24.5818672Z 
2022-05-18T04:03:24.5818795Z Running tests...
2022-05-18T04:03:24.5819382Z ----------------------------------------------------------------------
2022-05-18T04:03:24.8997736Z   test_profiler_rpc_key_names (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26376
2022-05-18T04:03:24.9021586Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26377
2022-05-18T04:03:24.9045103Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26378
2022-05-18T04:03:24.9069095Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 26379
2022-05-18T04:03:25.5914362Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzdk3fw80
2022-05-18T04:03:25.5915121Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzdk3fw80/_remote_module_non_scriptable.py
2022-05-18T04:03:25.6324509Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8yy32a9v
2022-05-18T04:03:25.6325289Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8yy32a9v/_remote_module_non_scriptable.py
2022-05-18T04:03:25.7440413Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0iqljkud
2022-05-18T04:03:25.7441152Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0iqljkud/_remote_module_non_scriptable.py
2022-05-18T04:03:25.7518155Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwt7f132y
2022-05-18T04:03:25.7520830Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwt7f132y/_remote_module_non_scriptable.py
2022-05-18T04:03:25.8443385Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:03:25.8804056Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:03:25.9951270Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:03:26.0015250Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:03:26.5111970Z ok (1.929s)
2022-05-18T04:03:26.5112243Z 
2022-05-18T04:03:26.5112753Z ----------------------------------------------------------------------
2022-05-18T04:03:26.5113085Z Ran 1 test in 1.929s
2022-05-18T04:03:26.5113203Z 
2022-05-18T04:03:26.5113264Z OK
2022-05-18T04:03:26.5113354Z 
2022-05-18T04:03:26.5113448Z Generating XML reports...
2022-05-18T04:03:26.5147406Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040324.xml
2022-05-18T04:03:27.2849717Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl6v3zuia
2022-05-18T04:03:27.2850671Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl6v3zuia/_remote_module_non_scriptable.py
2022-05-18T04:03:27.5419362Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:03:27.5429909Z 
2022-05-18T04:03:27.5429991Z Running tests...
2022-05-18T04:03:27.5430996Z ----------------------------------------------------------------------
2022-05-18T04:03:27.8588887Z   test_profiler_rpc_memory (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26601
2022-05-18T04:03:27.8611679Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26602
2022-05-18T04:03:27.8634910Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26603
2022-05-18T04:03:27.8659183Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 26604
2022-05-18T04:03:28.4908506Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmper9s9xvw
2022-05-18T04:03:28.4909342Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmper9s9xvw/_remote_module_non_scriptable.py
2022-05-18T04:03:28.4934761Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5gayxurw
2022-05-18T04:03:28.4936769Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5gayxurw/_remote_module_non_scriptable.py
2022-05-18T04:03:28.5074445Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8lc2ax2p
2022-05-18T04:03:28.5075580Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8lc2ax2p/_remote_module_non_scriptable.py
2022-05-18T04:03:28.5218077Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp48blcki5
2022-05-18T04:03:28.5218831Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp48blcki5/_remote_module_non_scriptable.py
2022-05-18T04:03:28.7430529Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:03:28.7450724Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:03:28.7621938Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:03:28.7751269Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:03:29.2700924Z ok (1.727s)
2022-05-18T04:03:29.2701183Z 
2022-05-18T04:03:29.2701736Z ----------------------------------------------------------------------
2022-05-18T04:03:29.2702147Z Ran 1 test in 1.727s
2022-05-18T04:03:29.2702314Z 
2022-05-18T04:03:29.2702414Z OK
2022-05-18T04:03:29.2702562Z 
2022-05-18T04:03:29.2702704Z Generating XML reports...
2022-05-18T04:03:29.2737013Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040327.xml
2022-05-18T04:03:30.0627892Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmposk9xxic
2022-05-18T04:03:30.0628914Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmposk9xxic/_remote_module_non_scriptable.py
2022-05-18T04:03:30.3185491Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:03:30.3195127Z 
2022-05-18T04:03:30.3195247Z Running tests...
2022-05-18T04:03:30.3195801Z ----------------------------------------------------------------------
2022-05-18T04:03:30.6416954Z   test_profiler_rpc_record_shapes (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26820
2022-05-18T04:03:30.6439861Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26821
2022-05-18T04:03:30.6463564Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26822
2022-05-18T04:03:30.6489269Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 26823
2022-05-18T04:03:31.3277988Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkwqhg49a
2022-05-18T04:03:31.3278914Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkwqhg49a/_remote_module_non_scriptable.py
2022-05-18T04:03:31.3709140Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpujh_gyn2
2022-05-18T04:03:31.3709936Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpujh_gyn2/_remote_module_non_scriptable.py
2022-05-18T04:03:31.4110866Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3f96u1km
2022-05-18T04:03:31.4112071Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3f96u1km/_remote_module_non_scriptable.py
2022-05-18T04:03:31.4304233Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr8jrgobu
2022-05-18T04:03:31.4304968Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr8jrgobu/_remote_module_non_scriptable.py
2022-05-18T04:03:31.5797938Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:03:31.6219119Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:03:31.6632097Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:03:31.6810790Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:03:32.1529865Z ok (1.833s)
2022-05-18T04:03:32.1530019Z 
2022-05-18T04:03:32.1530357Z ----------------------------------------------------------------------
2022-05-18T04:03:32.1530616Z Ran 1 test in 1.833s
2022-05-18T04:03:32.1530731Z 
2022-05-18T04:03:32.1530800Z OK
2022-05-18T04:03:32.1530950Z 
2022-05-18T04:03:32.1531045Z Generating XML reports...
2022-05-18T04:03:32.1564581Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040330.xml
2022-05-18T04:03:32.9110158Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsh1g_qzw
2022-05-18T04:03:32.9111049Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsh1g_qzw/_remote_module_non_scriptable.py
2022-05-18T04:03:33.1671788Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:03:33.1680970Z 
2022-05-18T04:03:33.1681105Z Running tests...
2022-05-18T04:03:33.1681531Z ----------------------------------------------------------------------
2022-05-18T04:03:33.4890167Z   test_profiler_with_async_rpc_builtin (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27039
2022-05-18T04:03:33.4913080Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27040
2022-05-18T04:03:33.4937045Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27041
2022-05-18T04:03:33.4961568Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 27042
2022-05-18T04:03:34.1734571Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb__uriga
2022-05-18T04:03:34.1735325Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb__uriga/_remote_module_non_scriptable.py
2022-05-18T04:03:34.1753305Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpot1wgf3h
2022-05-18T04:03:34.1754898Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpot1wgf3h/_remote_module_non_scriptable.py
2022-05-18T04:03:34.2042164Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnwjkhrrm
2022-05-18T04:03:34.2042996Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnwjkhrrm/_remote_module_non_scriptable.py
2022-05-18T04:03:34.2141951Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxdyxygzz
2022-05-18T04:03:34.2143095Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxdyxygzz/_remote_module_non_scriptable.py
2022-05-18T04:03:34.4238188Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:03:34.4240363Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:03:34.4534713Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:03:34.4632121Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:03:34.9001750Z ok (1.732s)
2022-05-18T04:03:34.9001989Z 
2022-05-18T04:03:34.9002538Z ----------------------------------------------------------------------
2022-05-18T04:03:34.9003056Z Ran 1 test in 1.732s
2022-05-18T04:03:34.9003173Z 
2022-05-18T04:03:34.9003236Z OK
2022-05-18T04:03:34.9003330Z 
2022-05-18T04:03:34.9003425Z Generating XML reports...
2022-05-18T04:03:34.9036607Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040333.xml
2022-05-18T04:03:35.6754598Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpof6cj85n
2022-05-18T04:03:35.6755653Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpof6cj85n/_remote_module_non_scriptable.py
2022-05-18T04:03:35.9282359Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:03:35.9291944Z 
2022-05-18T04:03:35.9292043Z Running tests...
2022-05-18T04:03:35.9292646Z ----------------------------------------------------------------------
2022-05-18T04:03:36.2457654Z   test_profiler_with_async_rpc_builtin_single_threaded (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27258
2022-05-18T04:03:36.2480686Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27259
2022-05-18T04:03:36.2503884Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27260
2022-05-18T04:03:36.2527575Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 27261
2022-05-18T04:03:36.9125736Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyxcw2t5l
2022-05-18T04:03:36.9126528Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyxcw2t5l/_remote_module_non_scriptable.py
2022-05-18T04:03:36.9217590Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppyh1xg9f
2022-05-18T04:03:36.9219323Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppyh1xg9f/_remote_module_non_scriptable.py
2022-05-18T04:03:36.9473736Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm9knat34
2022-05-18T04:03:36.9474495Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm9knat34/_remote_module_non_scriptable.py
2022-05-18T04:03:36.9702668Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmvbs2xql
2022-05-18T04:03:36.9703539Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmvbs2xql/_remote_module_non_scriptable.py
2022-05-18T04:03:37.1618127Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:03:37.1708319Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:03:37.1941008Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:03:37.2195429Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:03:37.6567652Z ok (1.727s)
2022-05-18T04:03:37.6567888Z 
2022-05-18T04:03:37.6568357Z ----------------------------------------------------------------------
2022-05-18T04:03:37.6568630Z Ran 1 test in 1.727s
2022-05-18T04:03:37.6568747Z 
2022-05-18T04:03:37.6568811Z OK
2022-05-18T04:03:37.6568903Z 
2022-05-18T04:03:37.6569005Z Generating XML reports...
2022-05-18T04:03:37.6603926Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040335.xml
2022-05-18T04:03:38.4241354Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp89ifrd0v
2022-05-18T04:03:38.4242063Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp89ifrd0v/_remote_module_non_scriptable.py
2022-05-18T04:03:38.6776318Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:03:38.6786213Z 
2022-05-18T04:03:38.6786607Z Running tests...
2022-05-18T04:03:38.6786998Z ----------------------------------------------------------------------
2022-05-18T04:03:38.9922201Z   test_profiler_with_async_rpc_udf (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27477
2022-05-18T04:03:38.9944715Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27478
2022-05-18T04:03:38.9967648Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27479
2022-05-18T04:03:38.9991502Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 27480
2022-05-18T04:03:39.6064653Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq11wye22
2022-05-18T04:03:39.6065415Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq11wye22/_remote_module_non_scriptable.py
2022-05-18T04:03:39.6225382Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppa48peqb
2022-05-18T04:03:39.6226930Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppa48peqb/_remote_module_non_scriptable.py
2022-05-18T04:03:39.6232408Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzmcubw1m
2022-05-18T04:03:39.6234464Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzmcubw1m/_remote_module_non_scriptable.py
2022-05-18T04:03:39.6241445Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4bxvudqt
2022-05-18T04:03:39.6243092Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4bxvudqt/_remote_module_non_scriptable.py
2022-05-18T04:03:39.8570737Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:03:39.8691603Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:03:39.8722063Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:03:39.8724328Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:03:42.0994250Z [W utils.cpp:176] Warning: Profiling a distributed call with the Kineto profiler will profile the caller, but not the worker. (function operator())
2022-05-18T04:03:43.4079496Z ok (4.729s)
2022-05-18T04:03:43.4079650Z 
2022-05-18T04:03:43.4079986Z ----------------------------------------------------------------------
2022-05-18T04:03:43.4080257Z Ran 1 test in 4.729s
2022-05-18T04:03:43.4080384Z 
2022-05-18T04:03:43.4080437Z OK
2022-05-18T04:03:43.4080583Z 
2022-05-18T04:03:43.4080675Z Generating XML reports...
2022-05-18T04:03:43.4114929Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040338.xml
2022-05-18T04:03:44.2018277Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp94ypin9x
2022-05-18T04:03:44.2019213Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp94ypin9x/_remote_module_non_scriptable.py
2022-05-18T04:03:44.4562075Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:03:44.4572212Z 
2022-05-18T04:03:44.4572631Z Running tests...
2022-05-18T04:03:44.4573289Z ----------------------------------------------------------------------
2022-05-18T04:03:44.7796332Z   test_profiler_with_async_rpc_udf_single_threaded (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27697
2022-05-18T04:03:44.7818146Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27698
2022-05-18T04:03:44.7841952Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27699
2022-05-18T04:03:44.7866675Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 27700
2022-05-18T04:03:45.3765502Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpatjxzi1d
2022-05-18T04:03:45.3766289Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpatjxzi1d/_remote_module_non_scriptable.py
2022-05-18T04:03:45.3836881Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu_49qq38
2022-05-18T04:03:45.3838203Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu_49qq38/_remote_module_non_scriptable.py
2022-05-18T04:03:45.3915332Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzm2l4sro
2022-05-18T04:03:45.3916281Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzm2l4sro/_remote_module_non_scriptable.py
2022-05-18T04:03:45.4461440Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz19dfww_
2022-05-18T04:03:45.4462246Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz19dfww_/_remote_module_non_scriptable.py
2022-05-18T04:03:45.6296293Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:03:45.6362157Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:03:45.6434758Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:03:45.6985501Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:03:47.9702229Z [W utils.cpp:176] Warning: Profiling a distributed call with the Kineto profiler will profile the caller, but not the worker. (function operator())
2022-05-18T04:03:49.1954547Z ok (4.738s)
2022-05-18T04:03:49.1954812Z 
2022-05-18T04:03:49.1955327Z ----------------------------------------------------------------------
2022-05-18T04:03:49.1955722Z Ran 1 test in 4.738s
2022-05-18T04:03:49.1955844Z 
2022-05-18T04:03:49.1955911Z OK
2022-05-18T04:03:49.1956007Z 
2022-05-18T04:03:49.1956103Z Generating XML reports...
2022-05-18T04:03:49.1990365Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040344.xml
2022-05-18T04:03:49.9976946Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfnf5pm34
2022-05-18T04:03:49.9977821Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfnf5pm34/_remote_module_non_scriptable.py
2022-05-18T04:03:50.2531263Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:03:50.2540284Z 
2022-05-18T04:03:50.2540502Z Running tests...
2022-05-18T04:03:50.2540925Z ----------------------------------------------------------------------
2022-05-18T04:03:50.5752787Z   test_profiler_with_autograd_context (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27917
2022-05-18T04:03:50.5776336Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27918
2022-05-18T04:03:50.5800130Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27919
2022-05-18T04:03:50.5824337Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 27920
2022-05-18T04:03:51.2803500Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp08bnsuyu
2022-05-18T04:03:51.2804733Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp08bnsuyu/_remote_module_non_scriptable.py
2022-05-18T04:03:51.2905993Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp230zjs6g
2022-05-18T04:03:51.2907294Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp230zjs6g/_remote_module_non_scriptable.py
2022-05-18T04:03:51.2930276Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpms6z9e3y
2022-05-18T04:03:51.2932604Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpms6z9e3y/_remote_module_non_scriptable.py
2022-05-18T04:03:51.2966152Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuqwvfzfa
2022-05-18T04:03:51.2967924Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuqwvfzfa/_remote_module_non_scriptable.py
2022-05-18T04:03:51.5340069Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:03:51.5393198Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:03:51.5437889Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:03:51.5485286Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:03:51.9864533Z ok (1.732s)
2022-05-18T04:03:51.9864806Z 
2022-05-18T04:03:51.9865324Z ----------------------------------------------------------------------
2022-05-18T04:03:51.9865769Z Ran 1 test in 1.732s
2022-05-18T04:03:51.9865895Z 
2022-05-18T04:03:51.9865948Z OK
2022-05-18T04:03:51.9866040Z 
2022-05-18T04:03:51.9866134Z Generating XML reports...
2022-05-18T04:03:51.9899505Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040350.xml
2022-05-18T04:03:52.7810546Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6wd7sbsr
2022-05-18T04:03:52.7811535Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6wd7sbsr/_remote_module_non_scriptable.py
2022-05-18T04:03:53.0359829Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:03:53.0369904Z 
2022-05-18T04:03:53.0369990Z Running tests...
2022-05-18T04:03:53.0370844Z ----------------------------------------------------------------------
2022-05-18T04:03:53.3598406Z   test_profiler_with_autograd_context_single_threaded (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28136
2022-05-18T04:03:53.3620958Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28137
2022-05-18T04:03:53.3644976Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28138
2022-05-18T04:03:53.3670161Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 28139
2022-05-18T04:03:54.0839722Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeyj4lkfz
2022-05-18T04:03:54.0840499Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeyj4lkfz/_remote_module_non_scriptable.py
2022-05-18T04:03:54.1064986Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz29icz78
2022-05-18T04:03:54.1066141Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz29icz78/_remote_module_non_scriptable.py
2022-05-18T04:03:54.1112498Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp14enps73
2022-05-18T04:03:54.1113378Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp14enps73/_remote_module_non_scriptable.py
2022-05-18T04:03:54.1355049Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3caou23y
2022-05-18T04:03:54.1356260Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3caou23y/_remote_module_non_scriptable.py
2022-05-18T04:03:54.3377309Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:03:54.3569174Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:03:54.3629468Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:03:54.3856375Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:03:54.8739335Z ok (1.837s)
2022-05-18T04:03:54.8739586Z 
2022-05-18T04:03:54.8740120Z ----------------------------------------------------------------------
2022-05-18T04:03:54.8740500Z Ran 1 test in 1.837s
2022-05-18T04:03:54.8740621Z 
2022-05-18T04:03:54.8740669Z OK
2022-05-18T04:03:54.8740760Z 
2022-05-18T04:03:54.8740858Z Generating XML reports...
2022-05-18T04:03:54.8775127Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040353.xml
2022-05-18T04:03:55.6586167Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7wapjeu6
2022-05-18T04:03:55.6586932Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7wapjeu6/_remote_module_non_scriptable.py
2022-05-18T04:03:55.9113705Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:03:55.9123833Z 
2022-05-18T04:03:55.9124282Z Running tests...
2022-05-18T04:03:55.9124922Z ----------------------------------------------------------------------
2022-05-18T04:03:56.2252439Z   test_profiler_with_remote_builtin (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28355
2022-05-18T04:03:56.2276076Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28356
2022-05-18T04:03:56.2299029Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28357
2022-05-18T04:03:56.2324379Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 28358
2022-05-18T04:03:56.8906963Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpafr8rt_g
2022-05-18T04:03:56.8907759Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpafr8rt_g/_remote_module_non_scriptable.py
2022-05-18T04:03:56.9246521Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbhjcq5lu
2022-05-18T04:03:56.9247598Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbhjcq5lu/_remote_module_non_scriptable.py
2022-05-18T04:03:56.9321011Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplm_2fqzy
2022-05-18T04:03:56.9322123Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplm_2fqzy/_remote_module_non_scriptable.py
2022-05-18T04:03:56.9436919Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7u9zivny
2022-05-18T04:03:56.9437904Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7u9zivny/_remote_module_non_scriptable.py
2022-05-18T04:03:57.1461061Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:03:57.1802551Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:03:57.1891523Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:03:57.1968576Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:03:57.7365791Z ok (1.824s)
2022-05-18T04:03:57.7366038Z 
2022-05-18T04:03:57.7366546Z ----------------------------------------------------------------------
2022-05-18T04:03:57.7366914Z Ran 1 test in 1.824s
2022-05-18T04:03:57.7367033Z 
2022-05-18T04:03:57.7367096Z OK
2022-05-18T04:03:57.7367188Z 
2022-05-18T04:03:57.7367284Z Generating XML reports...
2022-05-18T04:03:57.7402369Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040355.xml
2022-05-18T04:03:58.5326698Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbkjnm7nt
2022-05-18T04:03:58.5327591Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbkjnm7nt/_remote_module_non_scriptable.py
2022-05-18T04:03:58.7892473Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:03:58.7902447Z 
2022-05-18T04:03:58.7902568Z Running tests...
2022-05-18T04:03:58.7903203Z ----------------------------------------------------------------------
2022-05-18T04:03:59.1180210Z   test_profiler_with_remote_builtin_single_threaded (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28574
2022-05-18T04:03:59.1204789Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28575
2022-05-18T04:03:59.1228042Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28576
2022-05-18T04:03:59.1253050Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 28577
2022-05-18T04:03:59.7155244Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp8akjnrd
2022-05-18T04:03:59.7156033Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp8akjnrd/_remote_module_non_scriptable.py
2022-05-18T04:03:59.7559088Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp004g6i9j
2022-05-18T04:03:59.7560123Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp004g6i9j/_remote_module_non_scriptable.py
2022-05-18T04:03:59.7669491Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps6rjkskt
2022-05-18T04:03:59.7670480Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps6rjkskt/_remote_module_non_scriptable.py
2022-05-18T04:03:59.7691578Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8fbdas5g
2022-05-18T04:03:59.7693370Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8fbdas5g/_remote_module_non_scriptable.py
2022-05-18T04:03:59.9724152Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:04:00.0146251Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:04:00.0225681Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:04:00.0256903Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:04:00.5293923Z ok (1.739s)
2022-05-18T04:04:00.5294088Z 
2022-05-18T04:04:00.5294429Z ----------------------------------------------------------------------
2022-05-18T04:04:00.5294673Z Ran 1 test in 1.739s
2022-05-18T04:04:00.5294790Z 
2022-05-18T04:04:00.5294853Z OK
2022-05-18T04:04:00.5294948Z 
2022-05-18T04:04:00.5295042Z Generating XML reports...
2022-05-18T04:04:00.5329215Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040358.xml
2022-05-18T04:04:01.3115951Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7q292w0f
2022-05-18T04:04:01.3116781Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7q292w0f/_remote_module_non_scriptable.py
2022-05-18T04:04:01.5672312Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:04:01.5682094Z 
2022-05-18T04:04:01.5682200Z Running tests...
2022-05-18T04:04:01.5682628Z ----------------------------------------------------------------------
2022-05-18T04:04:01.8838783Z   test_profiler_with_remote_udf (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28793
2022-05-18T04:04:01.8861668Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28794
2022-05-18T04:04:01.8885184Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28795
2022-05-18T04:04:01.8909710Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 28796
2022-05-18T04:04:02.5473764Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpraz8ng8c
2022-05-18T04:04:02.5474600Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpraz8ng8c/_remote_module_non_scriptable.py
2022-05-18T04:04:02.5669580Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpazcw4qpd
2022-05-18T04:04:02.5671250Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpazcw4qpd/_remote_module_non_scriptable.py
2022-05-18T04:04:02.5896469Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu6x3zqly
2022-05-18T04:04:02.5897462Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu6x3zqly/_remote_module_non_scriptable.py
2022-05-18T04:04:02.6303593Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfmdoa9xi
2022-05-18T04:04:02.6304339Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfmdoa9xi/_remote_module_non_scriptable.py
2022-05-18T04:04:02.7978517Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:04:02.8150436Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:04:02.8386006Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:04:02.8765826Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:04:06.3998591Z ok (4.831s)
2022-05-18T04:04:06.3998864Z 
2022-05-18T04:04:06.3999326Z ----------------------------------------------------------------------
2022-05-18T04:04:06.3999568Z Ran 1 test in 4.832s
2022-05-18T04:04:06.3999684Z 
2022-05-18T04:04:06.3999747Z OK
2022-05-18T04:04:06.3999842Z 
2022-05-18T04:04:06.3999938Z Generating XML reports...
2022-05-18T04:04:06.4034878Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040401.xml
2022-05-18T04:04:07.1750736Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuwcm2lm2
2022-05-18T04:04:07.1751568Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuwcm2lm2/_remote_module_non_scriptable.py
2022-05-18T04:04:07.4306759Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:04:07.4316651Z 
2022-05-18T04:04:07.4316992Z Running tests...
2022-05-18T04:04:07.4317632Z ----------------------------------------------------------------------
2022-05-18T04:04:07.7478929Z   test_profiler_with_remote_udf_single_threaded (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29012
2022-05-18T04:04:07.7500819Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29013
2022-05-18T04:04:07.7525010Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29014
2022-05-18T04:04:07.7549530Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 29015
2022-05-18T04:04:08.4090254Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp88ymr9ox
2022-05-18T04:04:08.4091410Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp88ymr9ox/_remote_module_non_scriptable.py
2022-05-18T04:04:08.4289142Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpinzdvb3f
2022-05-18T04:04:08.4290021Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpinzdvb3f/_remote_module_non_scriptable.py
2022-05-18T04:04:08.4704309Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi77fmips
2022-05-18T04:04:08.4705075Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi77fmips/_remote_module_non_scriptable.py
2022-05-18T04:04:08.5036681Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9ekbodxh
2022-05-18T04:04:08.5037926Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9ekbodxh/_remote_module_non_scriptable.py
2022-05-18T04:04:08.6594080Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:04:08.6769155Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:04:08.7192287Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:04:08.7529970Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:04:12.2704933Z ok (4.838s)
2022-05-18T04:04:12.2705192Z 
2022-05-18T04:04:12.2705726Z ----------------------------------------------------------------------
2022-05-18T04:04:12.2706082Z Ran 1 test in 4.839s
2022-05-18T04:04:12.2706187Z 
2022-05-18T04:04:12.2706250Z OK
2022-05-18T04:04:12.2706343Z 
2022-05-18T04:04:12.2706442Z Generating XML reports...
2022-05-18T04:04:12.2739861Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040407.xml
2022-05-18T04:04:13.0505102Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmvlh6bf0
2022-05-18T04:04:13.0505563Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmvlh6bf0/_remote_module_non_scriptable.py
2022-05-18T04:04:13.3085412Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:04:13.3095581Z 
2022-05-18T04:04:13.3096164Z Running tests...
2022-05-18T04:04:13.3096823Z ----------------------------------------------------------------------
2022-05-18T04:04:13.6258918Z   test_profiler_with_script_async_rpc (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29231
2022-05-18T04:04:13.6282167Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29232
2022-05-18T04:04:13.6305088Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29233
2022-05-18T04:04:13.6329304Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 29234
2022-05-18T04:04:14.2902240Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp6wkfn7o
2022-05-18T04:04:14.2903156Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp6wkfn7o/_remote_module_non_scriptable.py
2022-05-18T04:04:14.3348975Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwpomm861
2022-05-18T04:04:14.3350653Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwpomm861/_remote_module_non_scriptable.py
2022-05-18T04:04:14.3374287Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzb_fgawp
2022-05-18T04:04:14.3376198Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzb_fgawp/_remote_module_non_scriptable.py
2022-05-18T04:04:14.3520394Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp97i_c_wg
2022-05-18T04:04:14.3521707Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp97i_c_wg/_remote_module_non_scriptable.py
2022-05-18T04:04:14.5395303Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:04:14.5841503Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:04:14.5862688Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:04:14.5985583Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:04:15.0371292Z ok (1.727s)
2022-05-18T04:04:15.0371423Z 
2022-05-18T04:04:15.0371825Z ----------------------------------------------------------------------
2022-05-18T04:04:15.0372136Z Ran 1 test in 1.727s
2022-05-18T04:04:15.0372254Z 
2022-05-18T04:04:15.0372304Z OK
2022-05-18T04:04:15.0372396Z 
2022-05-18T04:04:15.0372496Z Generating XML reports...
2022-05-18T04:04:15.0405976Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040413.xml
2022-05-18T04:04:15.8076160Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt8f6s6nx
2022-05-18T04:04:15.8077175Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt8f6s6nx/_remote_module_non_scriptable.py
2022-05-18T04:04:16.0589403Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:04:16.0599224Z 
2022-05-18T04:04:16.0599327Z Running tests...
2022-05-18T04:04:16.0600333Z ----------------------------------------------------------------------
2022-05-18T04:04:16.3752356Z   test_profiler_with_script_async_rpc_single_threaded (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29450
2022-05-18T04:04:16.3776902Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29451
2022-05-18T04:04:16.3799504Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29452
2022-05-18T04:04:16.3823729Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 29453
2022-05-18T04:04:17.0551127Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpczr7hw3b
2022-05-18T04:04:17.0552101Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpczr7hw3b/_remote_module_non_scriptable.py
2022-05-18T04:04:17.0781039Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx6jrr2jp
2022-05-18T04:04:17.0782375Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx6jrr2jp/_remote_module_non_scriptable.py
2022-05-18T04:04:17.0833238Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5uom1ubn
2022-05-18T04:04:17.0835146Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5uom1ubn/_remote_module_non_scriptable.py
2022-05-18T04:04:17.0907854Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7f5573bz
2022-05-18T04:04:17.0909340Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7f5573bz/_remote_module_non_scriptable.py
2022-05-18T04:04:17.3037092Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:04:17.3270459Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:04:17.3303106Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:04:17.3389164Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:04:17.7864621Z ok (1.726s)
2022-05-18T04:04:17.7864851Z 
2022-05-18T04:04:17.7865428Z ----------------------------------------------------------------------
2022-05-18T04:04:17.7865905Z Ran 1 test in 1.726s
2022-05-18T04:04:17.7866097Z 
2022-05-18T04:04:17.7866199Z OK
2022-05-18T04:04:17.7866293Z 
2022-05-18T04:04:17.7866391Z Generating XML reports...
2022-05-18T04:04:17.7900048Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040416.xml
2022-05-18T04:04:18.5558049Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbl4wy7lu
2022-05-18T04:04:18.5558709Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbl4wy7lu/_remote_module_non_scriptable.py
2022-05-18T04:04:18.8080287Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:04:18.8089518Z 
2022-05-18T04:04:18.8089631Z Running tests...
2022-05-18T04:04:18.8090864Z ----------------------------------------------------------------------
2022-05-18T04:04:19.1243093Z   test_profiler_with_script_remote_rpc (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29669
2022-05-18T04:04:19.1265021Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29670
2022-05-18T04:04:19.1288974Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29671
2022-05-18T04:04:19.1312948Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 29672
2022-05-18T04:04:19.7641053Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8pi5y0ao
2022-05-18T04:04:19.7641778Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8pi5y0ao/_remote_module_non_scriptable.py
2022-05-18T04:04:19.7678978Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9zqtp7v3
2022-05-18T04:04:19.7680950Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbd6437vd
2022-05-18T04:04:19.7681609Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9zqtp7v3/_remote_module_non_scriptable.py
2022-05-18T04:04:19.7683437Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbd6437vd/_remote_module_non_scriptable.py
2022-05-18T04:04:19.7770287Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsjsnxp7c
2022-05-18T04:04:19.7772428Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsjsnxp7c/_remote_module_non_scriptable.py
2022-05-18T04:04:20.0106722Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:04:20.0149639Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:04:20.0165402Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:04:20.0247891Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:04:20.5353366Z ok (1.726s)
2022-05-18T04:04:20.5353597Z 
2022-05-18T04:04:20.5354139Z ----------------------------------------------------------------------
2022-05-18T04:04:20.5354480Z Ran 1 test in 1.726s
2022-05-18T04:04:20.5354582Z 
2022-05-18T04:04:20.5354642Z OK
2022-05-18T04:04:20.5354732Z 
2022-05-18T04:04:20.5354831Z Generating XML reports...
2022-05-18T04:04:20.5388882Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040418.xml
2022-05-18T04:04:21.3023816Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8yhn4gzk
2022-05-18T04:04:21.3024574Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8yhn4gzk/_remote_module_non_scriptable.py
2022-05-18T04:04:21.5547601Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:04:21.5557781Z 
2022-05-18T04:04:21.5558236Z Running tests...
2022-05-18T04:04:21.5558642Z ----------------------------------------------------------------------
2022-05-18T04:04:21.8699071Z   test_profiler_with_script_remote_rpc_single_threaded (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29888
2022-05-18T04:04:21.8721469Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29889
2022-05-18T04:04:21.8744770Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29890
2022-05-18T04:04:21.8768897Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 29891
2022-05-18T04:04:22.4485397Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzga_gif_
2022-05-18T04:04:22.4487237Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzga_gif_/_remote_module_non_scriptable.py
2022-05-18T04:04:22.4730464Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv14t_tiu
2022-05-18T04:04:22.4731330Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv14t_tiu/_remote_module_non_scriptable.py
2022-05-18T04:04:22.5063874Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2joaqs9y
2022-05-18T04:04:22.5064872Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2joaqs9y/_remote_module_non_scriptable.py
2022-05-18T04:04:22.5235401Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsw2h0a3x
2022-05-18T04:04:22.5236197Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsw2h0a3x/_remote_module_non_scriptable.py
2022-05-18T04:04:22.6981897Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:04:22.7199370Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:04:22.7554022Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:04:22.7729633Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:04:23.2808666Z ok (1.725s)
2022-05-18T04:04:23.2808840Z 
2022-05-18T04:04:23.2809209Z ----------------------------------------------------------------------
2022-05-18T04:04:23.2809533Z Ran 1 test in 1.725s
2022-05-18T04:04:23.2809651Z 
2022-05-18T04:04:23.2809700Z OK
2022-05-18T04:04:23.2809793Z 
2022-05-18T04:04:23.2809888Z Generating XML reports...
2022-05-18T04:04:23.2844052Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040421.xml
2022-05-18T04:04:24.0566272Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp02t3tk7t
2022-05-18T04:04:24.0567581Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp02t3tk7t/_remote_module_non_scriptable.py
2022-05-18T04:04:24.3091547Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:04:24.3100692Z 
2022-05-18T04:04:24.3100779Z Running tests...
2022-05-18T04:04:24.3101802Z ----------------------------------------------------------------------
2022-05-18T04:04:24.6237141Z   test_profiler_with_script_sync_rpc (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30107
2022-05-18T04:04:24.6259146Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30108
2022-05-18T04:04:24.6283096Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30109
2022-05-18T04:04:24.6306890Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 30110
2022-05-18T04:04:25.2831471Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl8etp319
2022-05-18T04:04:25.2832237Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl8etp319/_remote_module_non_scriptable.py
2022-05-18T04:04:25.2981137Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1_gkn6ec
2022-05-18T04:04:25.2982083Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1_gkn6ec/_remote_module_non_scriptable.py
2022-05-18T04:04:25.3126386Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf8iqt9i5
2022-05-18T04:04:25.3127283Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf8iqt9i5/_remote_module_non_scriptable.py
2022-05-18T04:04:25.3182738Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvh7acn6f
2022-05-18T04:04:25.3184944Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvh7acn6f/_remote_module_non_scriptable.py
2022-05-18T04:04:25.5315522Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:04:25.5476950Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:04:25.5602589Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:04:25.5681824Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:04:26.0348279Z ok (1.724s)
2022-05-18T04:04:26.0348501Z 
2022-05-18T04:04:26.0348983Z ----------------------------------------------------------------------
2022-05-18T04:04:26.0349429Z Ran 1 test in 1.725s
2022-05-18T04:04:26.0349637Z 
2022-05-18T04:04:26.0349744Z OK
2022-05-18T04:04:26.0349919Z 
2022-05-18T04:04:26.0350083Z Generating XML reports...
2022-05-18T04:04:26.0383694Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040424.xml
2022-05-18T04:04:26.8676173Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx4eeotzh
2022-05-18T04:04:26.8676853Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx4eeotzh/_remote_module_non_scriptable.py
2022-05-18T04:04:27.1212428Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:04:27.1221368Z 
2022-05-18T04:04:27.1221590Z Running tests...
2022-05-18T04:04:27.1222157Z ----------------------------------------------------------------------
2022-05-18T04:04:27.4480695Z   test_profiler_with_script_sync_rpc_single_threaded (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30326
2022-05-18T04:04:27.4504831Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30327
2022-05-18T04:04:27.4527826Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30328
2022-05-18T04:04:27.4551763Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 30329
2022-05-18T04:04:28.0498503Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd_p_e0o5
2022-05-18T04:04:28.0501184Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcgr8tuwj
2022-05-18T04:04:28.0502073Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd_p_e0o5/_remote_module_non_scriptable.py
2022-05-18T04:04:28.0502784Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcgr8tuwj/_remote_module_non_scriptable.py
2022-05-18T04:04:28.0946771Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptui0w16w
2022-05-18T04:04:28.0947526Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptui0w16w/_remote_module_non_scriptable.py
2022-05-18T04:04:28.1038140Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkxiyd1iv
2022-05-18T04:04:28.1039404Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkxiyd1iv/_remote_module_non_scriptable.py
2022-05-18T04:04:28.3033972Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:04:28.3035386Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:04:28.3520302Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:04:28.3634786Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:04:28.8593394Z ok (1.737s)
2022-05-18T04:04:28.8593614Z 
2022-05-18T04:04:28.8594089Z ----------------------------------------------------------------------
2022-05-18T04:04:28.8594554Z Ran 1 test in 1.737s
2022-05-18T04:04:28.8594717Z 
2022-05-18T04:04:28.8594769Z OK
2022-05-18T04:04:28.8594862Z 
2022-05-18T04:04:28.8594957Z Generating XML reports...
2022-05-18T04:04:28.8627800Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040427.xml
2022-05-18T04:04:29.6405206Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnfjc2c92
2022-05-18T04:04:29.6405888Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnfjc2c92/_remote_module_non_scriptable.py
2022-05-18T04:04:29.9157814Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:04:29.9168211Z 
2022-05-18T04:04:29.9168471Z Running tests...
2022-05-18T04:04:29.9168872Z ----------------------------------------------------------------------
2022-05-18T04:04:30.2387757Z   test_profiler_with_sync_rpc_builtin (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30545
2022-05-18T04:04:30.2410653Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30546
2022-05-18T04:04:30.2434580Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30547
2022-05-18T04:04:30.2458667Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 30548
2022-05-18T04:04:30.8913808Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7izmrw3p
2022-05-18T04:04:30.8914695Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7izmrw3p/_remote_module_non_scriptable.py
2022-05-18T04:04:30.9104725Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyy2jvtgy
2022-05-18T04:04:30.9105461Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyy2jvtgy/_remote_module_non_scriptable.py
2022-05-18T04:04:30.9291695Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpliqw3_eg
2022-05-18T04:04:30.9292479Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpliqw3_eg/_remote_module_non_scriptable.py
2022-05-18T04:04:30.9380209Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp44dw6b8y
2022-05-18T04:04:30.9381164Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp44dw6b8y/_remote_module_non_scriptable.py
2022-05-18T04:04:31.1491962Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:04:31.1670991Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:04:31.1874974Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:04:31.1950950Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:04:31.6499482Z ok (1.733s)
2022-05-18T04:04:31.6499913Z 
2022-05-18T04:04:31.6500818Z ----------------------------------------------------------------------
2022-05-18T04:04:31.6501260Z Ran 1 test in 1.733s
2022-05-18T04:04:31.6501441Z 
2022-05-18T04:04:31.6501536Z OK
2022-05-18T04:04:31.6501698Z 
2022-05-18T04:04:31.6502146Z Generating XML reports...
2022-05-18T04:04:31.6534617Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040429.xml
2022-05-18T04:04:32.4184099Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq2l1tyf_
2022-05-18T04:04:32.4184581Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq2l1tyf_/_remote_module_non_scriptable.py
2022-05-18T04:04:32.6892030Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:04:32.6901706Z 
2022-05-18T04:04:32.6901850Z Running tests...
2022-05-18T04:04:32.6902572Z ----------------------------------------------------------------------
2022-05-18T04:04:33.0082910Z   test_profiler_with_sync_rpc_builtin_single_threaded (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30764
2022-05-18T04:04:33.0105963Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30765
2022-05-18T04:04:33.0130523Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30766
2022-05-18T04:04:33.0160805Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 30767
2022-05-18T04:04:33.6387980Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_5lnhl9n
2022-05-18T04:04:33.6389650Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_5lnhl9n/_remote_module_non_scriptable.py
2022-05-18T04:04:33.6521890Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpje05f5uv
2022-05-18T04:04:33.6523010Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpje05f5uv/_remote_module_non_scriptable.py
2022-05-18T04:04:33.6529310Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2950v9sm
2022-05-18T04:04:33.6532081Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2950v9sm/_remote_module_non_scriptable.py
2022-05-18T04:04:33.6674726Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuhitqaa8
2022-05-18T04:04:33.6677499Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuhitqaa8/_remote_module_non_scriptable.py
2022-05-18T04:04:33.8859980Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:04:33.9002138Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:04:33.9017173Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:04:33.9163454Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:04:34.4201108Z ok (1.730s)
2022-05-18T04:04:34.4201404Z 
2022-05-18T04:04:34.4201849Z ----------------------------------------------------------------------
2022-05-18T04:04:34.4202092Z Ran 1 test in 1.730s
2022-05-18T04:04:34.4202212Z 
2022-05-18T04:04:34.4202277Z OK
2022-05-18T04:04:34.4202373Z 
2022-05-18T04:04:34.4202468Z Generating XML reports...
2022-05-18T04:04:34.4236439Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040432.xml
2022-05-18T04:04:35.1892408Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplcyho8w_
2022-05-18T04:04:35.1893382Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplcyho8w_/_remote_module_non_scriptable.py
2022-05-18T04:04:35.4492294Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:04:35.4502109Z 
2022-05-18T04:04:35.4502588Z Running tests...
2022-05-18T04:04:35.4503234Z ----------------------------------------------------------------------
2022-05-18T04:04:35.7862836Z   test_profiler_with_sync_rpc_udf (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30983
2022-05-18T04:04:35.7887501Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30984
2022-05-18T04:04:35.7911052Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30985
2022-05-18T04:04:35.7936835Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 30986
2022-05-18T04:04:36.4566698Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkmipnp4i
2022-05-18T04:04:36.4567607Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmparjvgjng
2022-05-18T04:04:36.4568864Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkmipnp4i/_remote_module_non_scriptable.py
2022-05-18T04:04:36.4569843Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmparjvgjng/_remote_module_non_scriptable.py
2022-05-18T04:04:36.4673002Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpirsnxoym
2022-05-18T04:04:36.4674107Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpirsnxoym/_remote_module_non_scriptable.py
2022-05-18T04:04:36.4784980Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpasyxq7_0
2022-05-18T04:04:36.4786955Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpasyxq7_0/_remote_module_non_scriptable.py
2022-05-18T04:04:36.7250785Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:04:36.7263431Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:04:36.7328981Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:04:36.7466715Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:04:39.2007393Z ok (3.750s)
2022-05-18T04:04:39.2007648Z 
2022-05-18T04:04:39.2008126Z ----------------------------------------------------------------------
2022-05-18T04:04:39.2008496Z Ran 1 test in 3.750s
2022-05-18T04:04:39.2008681Z 
2022-05-18T04:04:39.2008780Z OK
2022-05-18T04:04:39.2008932Z 
2022-05-18T04:04:39.2009074Z Generating XML reports...
2022-05-18T04:04:39.2045532Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040435.xml
2022-05-18T04:04:39.9944537Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyd86vykl
2022-05-18T04:04:39.9945024Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyd86vykl/_remote_module_non_scriptable.py
2022-05-18T04:04:40.2464974Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:04:40.2474958Z 
2022-05-18T04:04:40.2475056Z Running tests...
2022-05-18T04:04:40.2475961Z ----------------------------------------------------------------------
2022-05-18T04:04:40.5672186Z   test_profiler_with_sync_rpc_udf_single_threaded (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31202
2022-05-18T04:04:40.5695119Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31203
2022-05-18T04:04:40.5718820Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31204
2022-05-18T04:04:40.5744235Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 31205
2022-05-18T04:04:41.1483625Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6y12iv7v
2022-05-18T04:04:41.1485574Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6y12iv7v/_remote_module_non_scriptable.py
2022-05-18T04:04:41.1697912Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8yu_hiy6
2022-05-18T04:04:41.1698675Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8yu_hiy6/_remote_module_non_scriptable.py
2022-05-18T04:04:41.1998457Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwircsvwz
2022-05-18T04:04:41.1999300Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwircsvwz/_remote_module_non_scriptable.py
2022-05-18T04:04:41.2179721Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm_r1psdc
2022-05-18T04:04:41.2180537Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm_r1psdc/_remote_module_non_scriptable.py
2022-05-18T04:04:41.4001523Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:04:41.4221774Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:04:41.4521084Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:04:41.4683759Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:04:43.8812689Z ok (3.633s)
2022-05-18T04:04:43.8812862Z 
2022-05-18T04:04:43.8813240Z ----------------------------------------------------------------------
2022-05-18T04:04:43.8813500Z Ran 1 test in 3.634s
2022-05-18T04:04:43.8813620Z 
2022-05-18T04:04:43.8813680Z OK
2022-05-18T04:04:43.8813770Z 
2022-05-18T04:04:43.8813911Z Generating XML reports...
2022-05-18T04:04:43.8846849Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040440.xml
2022-05-18T04:04:44.6562586Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0_0jt9vv
2022-05-18T04:04:44.6563483Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0_0jt9vv/_remote_module_non_scriptable.py
2022-05-18T04:04:44.9100409Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:04:44.9110422Z 
2022-05-18T04:04:44.9110536Z Running tests...
2022-05-18T04:04:44.9111091Z ----------------------------------------------------------------------
2022-05-18T04:04:45.2296824Z   test_py_built_in (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31421
2022-05-18T04:04:45.2320147Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31422
2022-05-18T04:04:45.2343529Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31423
2022-05-18T04:04:45.2368938Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 31424
2022-05-18T04:04:45.8538886Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpesxkzfji
2022-05-18T04:04:45.8540097Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpesxkzfji/_remote_module_non_scriptable.py
2022-05-18T04:04:45.8611129Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkrtaroev
2022-05-18T04:04:45.8612920Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkrtaroev/_remote_module_non_scriptable.py
2022-05-18T04:04:45.8882237Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvqf6ygki
2022-05-18T04:04:45.8883384Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjmn2xhry
2022-05-18T04:04:45.8884158Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvqf6ygki/_remote_module_non_scriptable.py
2022-05-18T04:04:45.8885487Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjmn2xhry/_remote_module_non_scriptable.py
2022-05-18T04:04:46.1023835Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:04:46.1069037Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:04:46.1364377Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:04:46.1365214Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:04:46.5405228Z ok (1.629s)
2022-05-18T04:04:46.5405412Z 
2022-05-18T04:04:46.5405751Z ----------------------------------------------------------------------
2022-05-18T04:04:46.5406026Z Ran 1 test in 1.629s
2022-05-18T04:04:46.5406140Z 
2022-05-18T04:04:46.5406201Z OK
2022-05-18T04:04:46.5406294Z 
2022-05-18T04:04:46.5406387Z Generating XML reports...
2022-05-18T04:04:46.5441103Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040444.xml
2022-05-18T04:04:47.3466450Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmputni605n
2022-05-18T04:04:47.3467229Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmputni605n/_remote_module_non_scriptable.py
2022-05-18T04:04:47.6015606Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:04:47.6025465Z 
2022-05-18T04:04:47.6025756Z Running tests...
2022-05-18T04:04:47.6026442Z ----------------------------------------------------------------------
2022-05-18T04:04:47.9174980Z   test_py_class_constructor (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31640
2022-05-18T04:04:47.9198724Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31641
2022-05-18T04:04:47.9221748Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31642
2022-05-18T04:04:47.9246009Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 31643
2022-05-18T04:04:48.5730211Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw4kjownl
2022-05-18T04:04:48.5731390Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw4kjownl/_remote_module_non_scriptable.py
2022-05-18T04:04:48.5878800Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptriyfw38
2022-05-18T04:04:48.5880018Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptriyfw38/_remote_module_non_scriptable.py
2022-05-18T04:04:48.5897334Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptv_pw2qi
2022-05-18T04:04:48.5899159Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptv_pw2qi/_remote_module_non_scriptable.py
2022-05-18T04:04:48.6127320Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr9_0ngws
2022-05-18T04:04:48.6128434Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr9_0ngws/_remote_module_non_scriptable.py
2022-05-18T04:04:48.8195929Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:04:48.8359296Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:04:48.8367132Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:04:48.8594904Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:04:49.3286385Z ok (1.726s)
2022-05-18T04:04:49.3286654Z 
2022-05-18T04:04:49.3287156Z ----------------------------------------------------------------------
2022-05-18T04:04:49.3287410Z Ran 1 test in 1.726s
2022-05-18T04:04:49.3287528Z 
2022-05-18T04:04:49.3287590Z OK
2022-05-18T04:04:49.3287685Z 
2022-05-18T04:04:49.3287778Z Generating XML reports...
2022-05-18T04:04:49.3321076Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040447.xml
2022-05-18T04:04:50.0995371Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5kp4c9bn
2022-05-18T04:04:50.0996119Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5kp4c9bn/_remote_module_non_scriptable.py
2022-05-18T04:04:50.3512239Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:04:50.3522304Z 
2022-05-18T04:04:50.3522567Z Running tests...
2022-05-18T04:04:50.6668289Z ----------------------------------------------------------------------
2022-05-18T04:04:50.6668766Z   test_py_class_instance_method (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31859
2022-05-18T04:04:50.6691107Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31860
2022-05-18T04:04:50.6715007Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31861
2022-05-18T04:04:50.6739200Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 31862
2022-05-18T04:04:51.3446392Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplczl8dqz
2022-05-18T04:04:51.3447202Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplczl8dqz/_remote_module_non_scriptable.py
2022-05-18T04:04:51.3572924Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5uwe8818
2022-05-18T04:04:51.3573671Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5uwe8818/_remote_module_non_scriptable.py
2022-05-18T04:04:51.3787332Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdwghbya6
2022-05-18T04:04:51.3788289Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdwghbya6/_remote_module_non_scriptable.py
2022-05-18T04:04:51.4056011Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplhsvqitc
2022-05-18T04:04:51.4057095Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplhsvqitc/_remote_module_non_scriptable.py
2022-05-18T04:04:51.5936303Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:04:51.6037544Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:04:51.6280057Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:04:51.6536533Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:04:52.0779862Z ok (1.725s)
2022-05-18T04:04:52.0780146Z 
2022-05-18T04:04:52.0780481Z ----------------------------------------------------------------------
2022-05-18T04:04:52.0780756Z Ran 1 test in 1.726s
2022-05-18T04:04:52.0780859Z 
2022-05-18T04:04:52.0780921Z OK
2022-05-18T04:04:52.0781012Z 
2022-05-18T04:04:52.0781105Z Generating XML reports...
2022-05-18T04:04:52.0815869Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040450.xml
2022-05-18T04:04:52.8487118Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpskvrww32
2022-05-18T04:04:52.8487733Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpskvrww32/_remote_module_non_scriptable.py
2022-05-18T04:04:53.1026846Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:04:53.1036017Z 
2022-05-18T04:04:53.1036197Z Running tests...
2022-05-18T04:04:53.1036581Z ----------------------------------------------------------------------
2022-05-18T04:04:53.4169758Z   test_py_class_method (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32078
2022-05-18T04:04:53.4192978Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32079
2022-05-18T04:04:53.4216802Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 32080
2022-05-18T04:04:53.4241164Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 32081
2022-05-18T04:04:54.1034693Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpajp7nqet
2022-05-18T04:04:54.1035716Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpajp7nqet/_remote_module_non_scriptable.py
2022-05-18T04:04:54.1170857Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9tufjjuo
2022-05-18T04:04:54.1172066Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9tufjjuo/_remote_module_non_scriptable.py
2022-05-18T04:04:54.1326189Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2nv7yk9w
2022-05-18T04:04:54.1327290Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2nv7yk9w/_remote_module_non_scriptable.py
2022-05-18T04:04:54.1516545Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp832y_md1
2022-05-18T04:04:54.1517386Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp832y_md1/_remote_module_non_scriptable.py
2022-05-18T04:04:54.3512543Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:04:54.3674474Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:04:54.3798646Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:04:54.3990741Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:04:54.8281253Z ok (1.724s)
2022-05-18T04:04:54.8281445Z 
2022-05-18T04:04:54.8281778Z ----------------------------------------------------------------------
2022-05-18T04:04:54.8282048Z Ran 1 test in 1.724s
2022-05-18T04:04:54.8282165Z 
2022-05-18T04:04:54.8282232Z OK
2022-05-18T04:04:54.8282325Z 
2022-05-18T04:04:54.8282423Z Generating XML reports...
2022-05-18T04:04:54.8315864Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040453.xml
2022-05-18T04:04:55.5999015Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1ygotaco
2022-05-18T04:04:55.5999821Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1ygotaco/_remote_module_non_scriptable.py
2022-05-18T04:04:55.8538225Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:04:55.8548057Z 
2022-05-18T04:04:55.8548155Z Running tests...
2022-05-18T04:04:55.8548881Z ----------------------------------------------------------------------
2022-05-18T04:04:56.1684090Z   test_py_class_static_method (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32297
2022-05-18T04:04:56.1708594Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32298
2022-05-18T04:04:56.1731768Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 32299
2022-05-18T04:04:56.1756412Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 32300
2022-05-18T04:04:56.8022469Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyry8jwk5
2022-05-18T04:04:56.8026208Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyry8jwk5/_remote_module_non_scriptable.py
2022-05-18T04:04:56.8705008Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbu6prqql
2022-05-18T04:04:56.8706097Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbu6prqql/_remote_module_non_scriptable.py
2022-05-18T04:04:56.8842209Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmcyzkl49
2022-05-18T04:04:56.8843322Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmcyzkl49/_remote_module_non_scriptable.py
2022-05-18T04:04:56.9094913Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5a9m4abb
2022-05-18T04:04:56.9095704Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5a9m4abb/_remote_module_non_scriptable.py
2022-05-18T04:04:57.0516341Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:04:57.1170594Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:04:57.1312100Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:04:57.1574981Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:04:57.5796207Z ok (1.725s)
2022-05-18T04:04:57.5796569Z 
2022-05-18T04:04:57.5797114Z ----------------------------------------------------------------------
2022-05-18T04:04:57.5797467Z Ran 1 test in 1.725s
2022-05-18T04:04:57.5797587Z 
2022-05-18T04:04:57.5797640Z OK
2022-05-18T04:04:57.5797731Z 
2022-05-18T04:04:57.5797825Z Generating XML reports...
2022-05-18T04:04:57.5831076Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040455.xml
2022-05-18T04:04:58.3433708Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeghwdrrf
2022-05-18T04:04:58.3434408Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeghwdrrf/_remote_module_non_scriptable.py
2022-05-18T04:04:58.5967056Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:04:58.5977376Z 
2022-05-18T04:04:58.5977834Z Running tests...
2022-05-18T04:04:58.5978228Z ----------------------------------------------------------------------
2022-05-18T04:04:58.9114491Z   test_py_function_exception (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32516
2022-05-18T04:04:58.9137441Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32517
2022-05-18T04:04:58.9161025Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 32518
2022-05-18T04:04:58.9185256Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 32519
2022-05-18T04:04:59.6090494Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8ickkv32
2022-05-18T04:04:59.6149232Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8ickkv32/_remote_module_non_scriptable.py
2022-05-18T04:04:59.6259103Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvznu8d2q
2022-05-18T04:04:59.6260313Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvznu8d2q/_remote_module_non_scriptable.py
2022-05-18T04:04:59.6518843Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqynkue7u
2022-05-18T04:04:59.6520294Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqynkue7u/_remote_module_non_scriptable.py
2022-05-18T04:04:59.6722211Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj0a9tyko
2022-05-18T04:04:59.6723370Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj0a9tyko/_remote_module_non_scriptable.py
2022-05-18T04:04:59.8587316Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:04:59.8745395Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:04:59.9007168Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:04:59.9189212Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:05:00.1372494Z On WorkerInfo(id=1, name=worker1):
2022-05-18T04:05:00.1373418Z TypeError('no_result() takes 0 positional arguments but 1 was given')
2022-05-18T04:05:00.1373986Z Traceback (most recent call last):
2022-05-18T04:05:00.1374850Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function
2022-05-18T04:05:00.1375556Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-05-18T04:05:00.1376179Z TypeError: no_result() takes 0 positional arguments but 1 was given
2022-05-18T04:05:00.1376529Z 
2022-05-18T04:05:00.1527032Z On WorkerInfo(id=2, name=worker2):
2022-05-18T04:05:00.1527865Z TypeError('no_result() takes 0 positional arguments but 1 was given')
2022-05-18T04:05:00.1528401Z Traceback (most recent call last):
2022-05-18T04:05:00.1529561Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function
2022-05-18T04:05:00.1530344Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-05-18T04:05:00.1530961Z TypeError: no_result() takes 0 positional arguments but 1 was given
2022-05-18T04:05:00.1531317Z 
2022-05-18T04:05:00.1570075Z On WorkerInfo(id=0, name=worker0):
2022-05-18T04:05:00.1570810Z TypeError('no_result() takes 0 positional arguments but 1 was given')
2022-05-18T04:05:00.1571266Z Traceback (most recent call last):
2022-05-18T04:05:00.1571936Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function
2022-05-18T04:05:00.1572513Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-05-18T04:05:00.1573024Z TypeError: no_result() takes 0 positional arguments but 1 was given
2022-05-18T04:05:00.1573310Z 
2022-05-18T04:05:00.1627845Z On WorkerInfo(id=3, name=worker3):
2022-05-18T04:05:00.1628675Z TypeError('no_result() takes 0 positional arguments but 1 was given')
2022-05-18T04:05:00.1629237Z Traceback (most recent call last):
2022-05-18T04:05:00.1630070Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function
2022-05-18T04:05:00.1630793Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-05-18T04:05:00.1631406Z TypeError: no_result() takes 0 positional arguments but 1 was given
2022-05-18T04:05:00.1631756Z 
2022-05-18T04:05:00.4227959Z ok (1.825s)
2022-05-18T04:05:00.4228184Z 
2022-05-18T04:05:00.4228635Z ----------------------------------------------------------------------
2022-05-18T04:05:00.4228892Z Ran 1 test in 1.825s
2022-05-18T04:05:00.4229009Z 
2022-05-18T04:05:00.4229070Z OK
2022-05-18T04:05:00.4229215Z 
2022-05-18T04:05:00.4229341Z Generating XML reports...
2022-05-18T04:05:00.4262514Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040458.xml
2022-05-18T04:05:01.2017004Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn1nieffy
2022-05-18T04:05:01.2017965Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn1nieffy/_remote_module_non_scriptable.py
2022-05-18T04:05:01.4575549Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:05:01.4585423Z 
2022-05-18T04:05:01.4585873Z Running tests...
2022-05-18T04:05:01.4586306Z ----------------------------------------------------------------------
2022-05-18T04:05:01.7785384Z   test_py_multi_async_call (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32735
2022-05-18T04:05:01.7807472Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32736
2022-05-18T04:05:01.7831160Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 32737
2022-05-18T04:05:01.7855583Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 32738
2022-05-18T04:05:02.4473899Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn57cyaod
2022-05-18T04:05:02.4474618Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn57cyaod/_remote_module_non_scriptable.py
2022-05-18T04:05:02.4610166Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmposigql49
2022-05-18T04:05:02.4612213Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmposigql49/_remote_module_non_scriptable.py
2022-05-18T04:05:02.5064949Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5dfzqxuo
2022-05-18T04:05:02.5065832Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5dfzqxuo/_remote_module_non_scriptable.py
2022-05-18T04:05:02.5170968Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpilguwghe
2022-05-18T04:05:02.5171941Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpilguwghe/_remote_module_non_scriptable.py
2022-05-18T04:05:02.6963350Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:05:02.7078857Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:05:02.7574045Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:05:02.7633085Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:05:03.1895198Z ok (1.731s)
2022-05-18T04:05:03.1895572Z 
2022-05-18T04:05:03.1896085Z ----------------------------------------------------------------------
2022-05-18T04:05:03.1896337Z Ran 1 test in 1.731s
2022-05-18T04:05:03.1896455Z 
2022-05-18T04:05:03.1896503Z OK
2022-05-18T04:05:03.1896621Z 
2022-05-18T04:05:03.1896718Z Generating XML reports...
2022-05-18T04:05:03.1930424Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040501.xml
2022-05-18T04:05:03.9637467Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy10j37qy
2022-05-18T04:05:03.9638188Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy10j37qy/_remote_module_non_scriptable.py
2022-05-18T04:05:04.2171417Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:05:04.2180541Z 
2022-05-18T04:05:04.2180640Z Running tests...
2022-05-18T04:05:04.2181100Z ----------------------------------------------------------------------
2022-05-18T04:05:04.5335775Z   test_py_nested_pickle (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 486
2022-05-18T04:05:04.5358258Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 487
2022-05-18T04:05:04.5381465Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 488
2022-05-18T04:05:04.5405917Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 489
2022-05-18T04:05:05.1998737Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9h6tw_13
2022-05-18T04:05:05.1999754Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9h6tw_13/_remote_module_non_scriptable.py
2022-05-18T04:05:05.2010635Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnwb1bcl7
2022-05-18T04:05:05.2012510Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnwb1bcl7/_remote_module_non_scriptable.py
2022-05-18T04:05:05.2666883Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpoms3b1jz
2022-05-18T04:05:05.2667593Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpoms3b1jz/_remote_module_non_scriptable.py
2022-05-18T04:05:05.2766106Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpua0phh1r
2022-05-18T04:05:05.2767045Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpua0phh1r/_remote_module_non_scriptable.py
2022-05-18T04:05:05.4476902Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:05:05.4504239Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:05:05.5155858Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:05:05.5231029Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:05:05.9446219Z ok (1.726s)
2022-05-18T04:05:05.9446426Z 
2022-05-18T04:05:05.9446766Z ----------------------------------------------------------------------
2022-05-18T04:05:05.9447022Z Ran 1 test in 1.726s
2022-05-18T04:05:05.9447144Z 
2022-05-18T04:05:05.9447206Z OK
2022-05-18T04:05:05.9447299Z 
2022-05-18T04:05:05.9447622Z Generating XML reports...
2022-05-18T04:05:05.9484293Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040504.xml
2022-05-18T04:05:06.7082091Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqb7qr087
2022-05-18T04:05:06.7082759Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqb7qr087/_remote_module_non_scriptable.py
2022-05-18T04:05:06.9612753Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:05:06.9621732Z 
2022-05-18T04:05:06.9621822Z Running tests...
2022-05-18T04:05:06.9622772Z ----------------------------------------------------------------------
2022-05-18T04:05:07.2796783Z   test_py_no_return_result (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 705
2022-05-18T04:05:07.2821124Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 706
2022-05-18T04:05:07.2844903Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 707
2022-05-18T04:05:07.2869250Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 708
2022-05-18T04:05:07.8970678Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9cz11w32
2022-05-18T04:05:07.8971459Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9cz11w32/_remote_module_non_scriptable.py
2022-05-18T04:05:07.9103372Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0uovlw4h
2022-05-18T04:05:07.9104496Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0uovlw4h/_remote_module_non_scriptable.py
2022-05-18T04:05:07.9290336Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvno6rf04
2022-05-18T04:05:07.9291091Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvno6rf04/_remote_module_non_scriptable.py
2022-05-18T04:05:07.9556842Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsq6phujg
2022-05-18T04:05:07.9557689Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsq6phujg/_remote_module_non_scriptable.py
2022-05-18T04:05:08.1495361Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:05:08.1595412Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:05:08.1790671Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:05:08.2032937Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:05:08.4367397Z do nothing
2022-05-18T04:05:08.4369392Z do nothing
2022-05-18T04:05:08.4522929Z do nothing
2022-05-18T04:05:08.4523349Z do nothing
2022-05-18T04:05:08.4563696Z do nothing
2022-05-18T04:05:08.4565996Z do nothing
2022-05-18T04:05:08.4647857Z do nothing
2022-05-18T04:05:08.4650599Z do nothing
2022-05-18T04:05:08.6910264Z ok (1.728s)
2022-05-18T04:05:08.6910479Z 
2022-05-18T04:05:08.6910933Z ----------------------------------------------------------------------
2022-05-18T04:05:08.6911341Z Ran 1 test in 1.729s
2022-05-18T04:05:08.6911535Z 
2022-05-18T04:05:08.6911628Z OK
2022-05-18T04:05:08.6911766Z 
2022-05-18T04:05:08.6911909Z Generating XML reports...
2022-05-18T04:05:08.6946287Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040506.xml
2022-05-18T04:05:09.4591193Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9_0qxw0z
2022-05-18T04:05:09.4592678Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9_0qxw0z/_remote_module_non_scriptable.py
2022-05-18T04:05:09.7112258Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:05:09.7122035Z 
2022-05-18T04:05:09.7122115Z Running tests...
2022-05-18T04:05:09.7122566Z ----------------------------------------------------------------------
2022-05-18T04:05:10.0293206Z   test_py_raise_in_user_func (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 924
2022-05-18T04:05:10.0316671Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 925
2022-05-18T04:05:10.0339715Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 926
2022-05-18T04:05:10.0364612Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 927
2022-05-18T04:05:10.6379237Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqofn90ys
2022-05-18T04:05:10.6380048Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqofn90ys/_remote_module_non_scriptable.py
2022-05-18T04:05:10.6507749Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzitwyar9
2022-05-18T04:05:10.6508787Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1rb3xi8m
2022-05-18T04:05:10.6509539Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzitwyar9/_remote_module_non_scriptable.py
2022-05-18T04:05:10.6510965Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1rb3xi8m/_remote_module_non_scriptable.py
2022-05-18T04:05:10.6689338Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphxmlqr9b
2022-05-18T04:05:10.6690058Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphxmlqr9b/_remote_module_non_scriptable.py
2022-05-18T04:05:10.8842916Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:05:10.8997516Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:05:10.8998026Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:05:10.9165700Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:05:11.1393858Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:05:11.1596573Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:05:11.1597470Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:05:11.1598281Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T04:05:11.1599555Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:05:11.1600699Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:05:11.1601864Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:05:11.1603013Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:05:11.4405550Z ok (1.728s)
2022-05-18T04:05:11.4405810Z 
2022-05-18T04:05:11.4406349Z ----------------------------------------------------------------------
2022-05-18T04:05:11.4406696Z Ran 1 test in 1.728s
2022-05-18T04:05:11.4406898Z 
2022-05-18T04:05:11.4407013Z OK
2022-05-18T04:05:11.4407186Z 
2022-05-18T04:05:11.4407358Z Generating XML reports...
2022-05-18T04:05:11.4440133Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040509.xml
2022-05-18T04:05:12.2069587Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp37bzaffm
2022-05-18T04:05:12.2070366Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp37bzaffm/_remote_module_non_scriptable.py
2022-05-18T04:05:12.4633507Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:05:12.4643228Z 
2022-05-18T04:05:12.4643804Z Running tests...
2022-05-18T04:05:12.4644335Z ----------------------------------------------------------------------
2022-05-18T04:05:12.7797916Z   test_py_raise_in_user_func_escaped_str (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1155
2022-05-18T04:05:12.7820963Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1156
2022-05-18T04:05:12.7844993Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1157
2022-05-18T04:05:12.7869408Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1158
2022-05-18T04:05:13.4659743Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu1net46r
2022-05-18T04:05:13.4661384Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu1net46r/_remote_module_non_scriptable.py
2022-05-18T04:05:13.4889069Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnb2p8hzv
2022-05-18T04:05:13.4889915Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnb2p8hzv/_remote_module_non_scriptable.py
2022-05-18T04:05:13.5051336Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpegdtajs1
2022-05-18T04:05:13.5052521Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpegdtajs1/_remote_module_non_scriptable.py
2022-05-18T04:05:13.5342473Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp90dsr5qk
2022-05-18T04:05:13.5343442Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp90dsr5qk/_remote_module_non_scriptable.py
2022-05-18T04:05:13.7149635Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:05:13.7369397Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:05:13.7519522Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:05:13.7817945Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:05:14.0048159Z On WorkerInfo(id=1, name=worker1):
2022-05-18T04:05:14.0049150Z ValueError('\nFirst line of error \n next line of error \n last line of error')
2022-05-18T04:05:14.0049731Z Traceback (most recent call last):
2022-05-18T04:05:14.0050642Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function
2022-05-18T04:05:14.0051370Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-05-18T04:05:14.0052369Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 404, in raise_func_escape
2022-05-18T04:05:14.0053044Z     raise ValueError(expected_err_escape)
2022-05-18T04:05:14.0053494Z ValueError: 
2022-05-18T04:05:14.0053869Z First line of error 
2022-05-18T04:05:14.0054242Z  next line of error 
2022-05-18T04:05:14.0054615Z  last line of error
2022-05-18T04:05:14.0054845Z 
2022-05-18T04:05:14.0205997Z On WorkerInfo(id=2, name=worker2):
2022-05-18T04:05:14.0206885Z ValueError('\nFirst line of error \n next line of error \n last line of error')
2022-05-18T04:05:14.0207497Z Traceback (most recent call last):
2022-05-18T04:05:14.0208391Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function
2022-05-18T04:05:14.0209123Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-05-18T04:05:14.0210098Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 404, in raise_func_escape
2022-05-18T04:05:14.0210859Z     raise ValueError(expected_err_escape)
2022-05-18T04:05:14.0211291Z ValueError: 
2022-05-18T04:05:14.0211643Z First line of error 
2022-05-18T04:05:14.0212029Z  next line of error 
2022-05-18T04:05:14.0212412Z  last line of error
2022-05-18T04:05:14.0212651Z 
2022-05-18T04:05:14.0249249Z On WorkerInfo(id=0, name=worker0):
2022-05-18T04:05:14.0249945Z ValueError('\nFirst line of error \n next line of error \n last line of error')
2022-05-18T04:05:14.0250751Z Traceback (most recent call last):
2022-05-18T04:05:14.0251542Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function
2022-05-18T04:05:14.0252110Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-05-18T04:05:14.0252907Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 404, in raise_func_escape
2022-05-18T04:05:14.0253452Z     raise ValueError(expected_err_escape)
2022-05-18T04:05:14.0253801Z ValueError: 
2022-05-18T04:05:14.0254086Z First line of error 
2022-05-18T04:05:14.0254401Z  next line of error 
2022-05-18T04:05:14.0254718Z  last line of error
2022-05-18T04:05:14.0254887Z 
2022-05-18T04:05:14.0308072Z On WorkerInfo(id=3, name=worker3):
2022-05-18T04:05:14.0308900Z ValueError('\nFirst line of error \n next line of error \n last line of error')
2022-05-18T04:05:14.0309495Z Traceback (most recent call last):
2022-05-18T04:05:14.0310363Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function
2022-05-18T04:05:14.0311089Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-05-18T04:05:14.0312097Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 404, in raise_func_escape
2022-05-18T04:05:14.0312803Z     raise ValueError(expected_err_escape)
2022-05-18T04:05:14.0313209Z ValueError: 
2022-05-18T04:05:14.0313578Z First line of error 
2022-05-18T04:05:14.0313975Z  next line of error 
2022-05-18T04:05:14.0314327Z  last line of error
2022-05-18T04:05:14.0314559Z 
2022-05-18T04:05:14.2910877Z ok (1.826s)
2022-05-18T04:05:14.2911347Z 
2022-05-18T04:05:14.2912174Z ----------------------------------------------------------------------
2022-05-18T04:05:14.2912498Z Ran 1 test in 1.827s
2022-05-18T04:05:14.2912625Z 
2022-05-18T04:05:14.2912689Z OK
2022-05-18T04:05:14.2912769Z 
2022-05-18T04:05:14.2912878Z Generating XML reports...
2022-05-18T04:05:14.2947769Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040512.xml
2022-05-18T04:05:15.0669313Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpal_49e0n
2022-05-18T04:05:15.0670072Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpal_49e0n/_remote_module_non_scriptable.py
2022-05-18T04:05:15.3201354Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:05:15.3210870Z 
2022-05-18T04:05:15.3211202Z Running tests...
2022-05-18T04:05:15.3211847Z ----------------------------------------------------------------------
2022-05-18T04:05:15.6364885Z   test_py_rpc_rref_args (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1374
2022-05-18T04:05:15.6386757Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1375
2022-05-18T04:05:15.6409755Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1376
2022-05-18T04:05:15.6434479Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1377
2022-05-18T04:05:16.2484193Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp93wum6i7
2022-05-18T04:05:16.2485270Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp93wum6i7/_remote_module_non_scriptable.py
2022-05-18T04:05:16.2554311Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplatcyh_9
2022-05-18T04:05:16.2556237Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplatcyh_9/_remote_module_non_scriptable.py
2022-05-18T04:05:16.2580378Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp3_szygi
2022-05-18T04:05:16.2582125Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp3_szygi/_remote_module_non_scriptable.py
2022-05-18T04:05:16.2613100Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf6q062ut
2022-05-18T04:05:16.2614715Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf6q062ut/_remote_module_non_scriptable.py
2022-05-18T04:05:16.4971817Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:05:16.5025046Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:05:16.5061062Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:05:16.5075066Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:05:17.0476126Z ok (1.726s)
2022-05-18T04:05:17.0476398Z 
2022-05-18T04:05:17.0476909Z ----------------------------------------------------------------------
2022-05-18T04:05:17.0477171Z Ran 1 test in 1.726s
2022-05-18T04:05:17.0477287Z 
2022-05-18T04:05:17.0477336Z OK
2022-05-18T04:05:17.0477430Z 
2022-05-18T04:05:17.0477545Z Generating XML reports...
2022-05-18T04:05:17.0511251Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040515.xml
2022-05-18T04:05:17.8430748Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpng52hk3m
2022-05-18T04:05:17.8431283Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpng52hk3m/_remote_module_non_scriptable.py
2022-05-18T04:05:18.1011901Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:05:18.1021653Z 
2022-05-18T04:05:18.1021763Z Running tests...
2022-05-18T04:05:18.1022204Z ----------------------------------------------------------------------
2022-05-18T04:05:18.4335596Z   test_py_rref_args (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1593
2022-05-18T04:05:18.4358838Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1594
2022-05-18T04:05:18.4382361Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1595
2022-05-18T04:05:18.4407015Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1596
2022-05-18T04:05:19.0263279Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqbhfftb_
2022-05-18T04:05:19.0265083Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqbhfftb_/_remote_module_non_scriptable.py
2022-05-18T04:05:19.0430451Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphodlva1f
2022-05-18T04:05:19.0431278Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphodlva1f/_remote_module_non_scriptable.py
2022-05-18T04:05:19.0719466Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0pg1446l
2022-05-18T04:05:19.0720205Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0pg1446l/_remote_module_non_scriptable.py
2022-05-18T04:05:19.0794226Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7gdbx2qf
2022-05-18T04:05:19.0795263Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7gdbx2qf/_remote_module_non_scriptable.py
2022-05-18T04:05:19.2759892Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:05:19.2938714Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:05:19.3244553Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:05:19.3305934Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:05:19.8448782Z ok (1.742s)
2022-05-18T04:05:19.8449040Z 
2022-05-18T04:05:19.8449469Z ----------------------------------------------------------------------
2022-05-18T04:05:19.8449736Z Ran 1 test in 1.743s
2022-05-18T04:05:19.8449852Z 
2022-05-18T04:05:19.8449900Z OK
2022-05-18T04:05:19.8449994Z 
2022-05-18T04:05:19.8450395Z Generating XML reports...
2022-05-18T04:05:19.8483786Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040518.xml
2022-05-18T04:05:20.6580088Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpay3__o0t
2022-05-18T04:05:20.6580819Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpay3__o0t/_remote_module_non_scriptable.py
2022-05-18T04:05:20.9127882Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:05:20.9137872Z 
2022-05-18T04:05:20.9137996Z Running tests...
2022-05-18T04:05:20.9138478Z ----------------------------------------------------------------------
2022-05-18T04:05:21.2348688Z   test_py_rref_args_user_share (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1812
2022-05-18T04:05:21.2373578Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1813
2022-05-18T04:05:21.2398341Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1814
2022-05-18T04:05:21.2425206Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1815
2022-05-18T04:05:21.8784485Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeeroiboa
2022-05-18T04:05:21.8785327Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeeroiboa/_remote_module_non_scriptable.py
2022-05-18T04:05:21.8942511Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7w_3mt62
2022-05-18T04:05:21.8944499Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7w_3mt62/_remote_module_non_scriptable.py
2022-05-18T04:05:21.8996781Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9ly6efhz
2022-05-18T04:05:21.8998346Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9ly6efhz/_remote_module_non_scriptable.py
2022-05-18T04:05:21.9126402Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmped1j4cti
2022-05-18T04:05:21.9127568Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmped1j4cti/_remote_module_non_scriptable.py
2022-05-18T04:05:22.1269874Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:05:22.1402740Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:05:22.1474356Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:05:22.1577563Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:05:22.6464507Z ok (1.732s)
2022-05-18T04:05:22.6464743Z 
2022-05-18T04:05:22.6465296Z ----------------------------------------------------------------------
2022-05-18T04:05:22.6465756Z Ran 1 test in 1.733s
2022-05-18T04:05:22.6465947Z 
2022-05-18T04:05:22.6466012Z OK
2022-05-18T04:05:22.6466092Z 
2022-05-18T04:05:22.6466193Z Generating XML reports...
2022-05-18T04:05:22.6499617Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040520.xml
2022-05-18T04:05:23.4158830Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv3du6odl
2022-05-18T04:05:23.4159877Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv3du6odl/_remote_module_non_scriptable.py
2022-05-18T04:05:23.6709269Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:05:23.6719184Z 
2022-05-18T04:05:23.6719279Z Running tests...
2022-05-18T04:05:23.6720261Z ----------------------------------------------------------------------
2022-05-18T04:05:23.9898224Z   test_py_tensors (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2031
2022-05-18T04:05:23.9921540Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2032
2022-05-18T04:05:23.9945009Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2033
2022-05-18T04:05:23.9969110Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2034
2022-05-18T04:05:24.6023449Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbg4dj6dv
2022-05-18T04:05:24.6024259Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbg4dj6dv/_remote_module_non_scriptable.py
2022-05-18T04:05:24.6028771Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsazgmhkp
2022-05-18T04:05:24.6031010Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsazgmhkp/_remote_module_non_scriptable.py
2022-05-18T04:05:24.6132382Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkdl44hnv
2022-05-18T04:05:24.6134135Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkdl44hnv/_remote_module_non_scriptable.py
2022-05-18T04:05:24.6445483Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp022vxgsv
2022-05-18T04:05:24.6446734Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp022vxgsv/_remote_module_non_scriptable.py
2022-05-18T04:05:24.8483116Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:05:24.8489680Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:05:24.8582201Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:05:24.8938392Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:05:25.3007095Z ok (1.628s)
2022-05-18T04:05:25.3007316Z 
2022-05-18T04:05:25.3007787Z ----------------------------------------------------------------------
2022-05-18T04:05:25.3008238Z Ran 1 test in 1.629s
2022-05-18T04:05:25.3008409Z 
2022-05-18T04:05:25.3008477Z OK
2022-05-18T04:05:25.3008571Z 
2022-05-18T04:05:25.3008666Z Generating XML reports...
2022-05-18T04:05:25.3043728Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040523.xml
2022-05-18T04:05:26.0760477Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpznlsxzgf
2022-05-18T04:05:26.0761244Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpznlsxzgf/_remote_module_non_scriptable.py
2022-05-18T04:05:26.3281715Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:05:26.3291489Z 
2022-05-18T04:05:26.3291957Z Running tests...
2022-05-18T04:05:26.3292602Z ----------------------------------------------------------------------
2022-05-18T04:05:26.6466987Z   test_py_tensors_in_container (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2250
2022-05-18T04:05:26.6489245Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2251
2022-05-18T04:05:26.6512810Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2252
2022-05-18T04:05:26.6537987Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2253
2022-05-18T04:05:27.2877757Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm3bjxah0
2022-05-18T04:05:27.2878540Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm3bjxah0/_remote_module_non_scriptable.py
2022-05-18T04:05:27.3061822Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj0nb_2qw
2022-05-18T04:05:27.3062581Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj0nb_2qw/_remote_module_non_scriptable.py
2022-05-18T04:05:27.3245468Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_floq6co
2022-05-18T04:05:27.3246224Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_floq6co/_remote_module_non_scriptable.py
2022-05-18T04:05:27.3559791Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvutuvgo1
2022-05-18T04:05:27.3560751Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvutuvgo1/_remote_module_non_scriptable.py
2022-05-18T04:05:27.5372596Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:05:27.5548443Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:05:27.5734918Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:05:27.6057722Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:05:28.0576832Z ok (1.728s)
2022-05-18T04:05:28.0577073Z 
2022-05-18T04:05:28.0577495Z ----------------------------------------------------------------------
2022-05-18T04:05:28.0577787Z Ran 1 test in 1.728s
2022-05-18T04:05:28.0577913Z 
2022-05-18T04:05:28.0578008Z OK
2022-05-18T04:05:28.0578097Z 
2022-05-18T04:05:28.0578196Z Generating XML reports...
2022-05-18T04:05:28.0612210Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040526.xml
2022-05-18T04:05:28.8323265Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp42v06u30
2022-05-18T04:05:28.8323764Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp42v06u30/_remote_module_non_scriptable.py
2022-05-18T04:05:29.0875590Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:05:29.0885213Z 
2022-05-18T04:05:29.0885331Z Running tests...
2022-05-18T04:05:29.0885930Z ----------------------------------------------------------------------
2022-05-18T04:05:29.4100776Z   test_py_tensors_multi_async_call (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2469
2022-05-18T04:05:29.4124595Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2470
2022-05-18T04:05:29.4148561Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2471
2022-05-18T04:05:29.4172600Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2472
2022-05-18T04:05:30.1202568Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprz49x3vn
2022-05-18T04:05:30.1205031Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprz49x3vn/_remote_module_non_scriptable.py
2022-05-18T04:05:30.1290503Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm5iyvidl
2022-05-18T04:05:30.1292056Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm5iyvidl/_remote_module_non_scriptable.py
2022-05-18T04:05:30.1719286Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsjtm6kkj
2022-05-18T04:05:30.1721546Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsjtm6kkj/_remote_module_non_scriptable.py
2022-05-18T04:05:30.2602006Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8kquuveh
2022-05-18T04:05:30.2603045Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8kquuveh/_remote_module_non_scriptable.py
2022-05-18T04:05:30.3683090Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:05:30.3753229Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:05:30.4440323Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:05:30.5081603Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:05:31.1217300Z ok (2.033s)
2022-05-18T04:05:31.1217528Z 
2022-05-18T04:05:31.1218078Z ----------------------------------------------------------------------
2022-05-18T04:05:31.1218348Z Ran 1 test in 2.033s
2022-05-18T04:05:31.1218462Z 
2022-05-18T04:05:31.1218526Z OK
2022-05-18T04:05:31.1218618Z 
2022-05-18T04:05:31.1218698Z Generating XML reports...
2022-05-18T04:05:31.1254145Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040529.xml
2022-05-18T04:05:31.9037461Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2cgg9u5g
2022-05-18T04:05:31.9038246Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2cgg9u5g/_remote_module_non_scriptable.py
2022-05-18T04:05:32.1575607Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:05:32.1585114Z 
2022-05-18T04:05:32.1585566Z Running tests...
2022-05-18T04:05:32.1585969Z ----------------------------------------------------------------------
2022-05-18T04:05:32.4755070Z   test_py_udf_remote (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2688
2022-05-18T04:05:32.4777874Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2689
2022-05-18T04:05:32.4801847Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2690
2022-05-18T04:05:32.4826477Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2691
2022-05-18T04:05:33.1582433Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0_qjhy35
2022-05-18T04:05:33.1583680Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0_qjhy35/_remote_module_non_scriptable.py
2022-05-18T04:05:33.1642159Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu7zwlyq6
2022-05-18T04:05:33.1643490Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu7zwlyq6/_remote_module_non_scriptable.py
2022-05-18T04:05:33.1824354Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph4ql13co
2022-05-18T04:05:33.1826298Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph4ql13co/_remote_module_non_scriptable.py
2022-05-18T04:05:33.2150906Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp500bmnz_
2022-05-18T04:05:33.2152418Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp500bmnz_/_remote_module_non_scriptable.py
2022-05-18T04:05:33.4071312Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:05:33.4118097Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:05:33.4303231Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:05:33.4642800Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:05:33.8866691Z ok (1.728s)
2022-05-18T04:05:33.8866926Z 
2022-05-18T04:05:33.8867322Z ----------------------------------------------------------------------
2022-05-18T04:05:33.8867608Z Ran 1 test in 1.728s
2022-05-18T04:05:33.8867732Z 
2022-05-18T04:05:33.8867795Z OK
2022-05-18T04:05:33.8867889Z 
2022-05-18T04:05:33.8868035Z Generating XML reports...
2022-05-18T04:05:33.8902188Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040532.xml
2022-05-18T04:05:34.6559434Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1xp0i1ie
2022-05-18T04:05:34.6559966Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1xp0i1ie/_remote_module_non_scriptable.py
2022-05-18T04:05:34.9090648Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:05:34.9100438Z 
2022-05-18T04:05:34.9100545Z Running tests...
2022-05-18T04:05:34.9101583Z ----------------------------------------------------------------------
2022-05-18T04:05:35.2243800Z   test_py_user_defined (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2907
2022-05-18T04:05:35.2266460Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2908
2022-05-18T04:05:35.2289581Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2909
2022-05-18T04:05:35.2314908Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2910
2022-05-18T04:05:35.8619446Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1jkx3egx
2022-05-18T04:05:35.8620686Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1jkx3egx/_remote_module_non_scriptable.py
2022-05-18T04:05:35.8827661Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj08knlwg
2022-05-18T04:05:35.8828403Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj08knlwg/_remote_module_non_scriptable.py
2022-05-18T04:05:35.9081959Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn77euvon
2022-05-18T04:05:35.9084508Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn77euvon/_remote_module_non_scriptable.py
2022-05-18T04:05:35.9085278Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi9d09dz3
2022-05-18T04:05:35.9086641Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi9d09dz3/_remote_module_non_scriptable.py
2022-05-18T04:05:36.1129466Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:05:36.1300259Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:05:36.1544580Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:05:36.1566365Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:05:36.6354921Z ok (1.725s)
2022-05-18T04:05:36.6355210Z 
2022-05-18T04:05:36.6355735Z ----------------------------------------------------------------------
2022-05-18T04:05:36.6355980Z Ran 1 test in 1.725s
2022-05-18T04:05:36.6356098Z 
2022-05-18T04:05:36.6356159Z OK
2022-05-18T04:05:36.6356251Z 
2022-05-18T04:05:36.6356348Z Generating XML reports...
2022-05-18T04:05:36.6389910Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040534.xml
2022-05-18T04:05:37.4162563Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu44ncn3e
2022-05-18T04:05:37.4163118Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu44ncn3e/_remote_module_non_scriptable.py
2022-05-18T04:05:37.6688166Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:05:37.6698169Z 
2022-05-18T04:05:37.6698560Z Running tests...
2022-05-18T04:05:37.6698966Z ----------------------------------------------------------------------
2022-05-18T04:05:37.9841846Z   test_register_rpc_backend_and_set_and_start_rpc_backend (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3126
2022-05-18T04:05:37.9864834Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3127
2022-05-18T04:05:37.9887773Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3128
2022-05-18T04:05:37.9911715Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3129
2022-05-18T04:05:38.6430426Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzy_0gd3z
2022-05-18T04:05:38.6431349Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzy_0gd3z/_remote_module_non_scriptable.py
2022-05-18T04:05:38.6495585Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp181n9zlm
2022-05-18T04:05:38.6497167Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp181n9zlm/_remote_module_non_scriptable.py
2022-05-18T04:05:38.6528237Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppxi2p489
2022-05-18T04:05:38.6530195Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppxi2p489/_remote_module_non_scriptable.py
2022-05-18T04:05:38.6632273Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpacloey3q
2022-05-18T04:05:38.6633250Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpacloey3q/_remote_module_non_scriptable.py
2022-05-18T04:05:38.8895736Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:05:38.8977364Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:05:38.9004185Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:05:38.9116316Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:05:39.0945905Z ok (1.424s)
2022-05-18T04:05:39.0946115Z 
2022-05-18T04:05:39.0946513Z ----------------------------------------------------------------------
2022-05-18T04:05:39.0946768Z Ran 1 test in 1.425s
2022-05-18T04:05:39.0946889Z 
2022-05-18T04:05:39.0946953Z OK
2022-05-18T04:05:39.0947045Z 
2022-05-18T04:05:39.0947139Z Generating XML reports...
2022-05-18T04:05:39.0980617Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040537.xml
2022-05-18T04:05:39.8184612Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwva312st
2022-05-18T04:05:39.8185350Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwva312st/_remote_module_non_scriptable.py
2022-05-18T04:05:40.0734598Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:05:40.0744330Z 
2022-05-18T04:05:40.0744468Z Running tests...
2022-05-18T04:05:40.0745074Z ----------------------------------------------------------------------
2022-05-18T04:05:40.3897752Z   test_reinit (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3181
2022-05-18T04:05:40.3921049Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3182
2022-05-18T04:05:40.3945581Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3183
2022-05-18T04:05:40.3969820Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3184
2022-05-18T04:05:40.9683444Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpiqn7n6ic
2022-05-18T04:05:40.9684221Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpiqn7n6ic/_remote_module_non_scriptable.py
2022-05-18T04:05:40.9928397Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz63uzf9k
2022-05-18T04:05:40.9929344Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz63uzf9k/_remote_module_non_scriptable.py
2022-05-18T04:05:41.0353658Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4787k2vy
2022-05-18T04:05:41.0354820Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4787k2vy/_remote_module_non_scriptable.py
2022-05-18T04:05:41.0451402Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmjg6jeer
2022-05-18T04:05:41.0452989Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmjg6jeer/_remote_module_non_scriptable.py
2022-05-18T04:05:41.2163506Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:05:41.2410712Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:05:41.2865871Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:05:41.2923095Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:05:41.5514113Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:05:41.5614753Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:05:41.5714484Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T04:05:41.5715436Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:05:41.5716587Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:05:41.5717578Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:05:41.5720358Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:05:41.5721230Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:05:41.8010742Z ok (1.726s)
2022-05-18T04:05:41.8010942Z 
2022-05-18T04:05:41.8011336Z ----------------------------------------------------------------------
2022-05-18T04:05:41.8011591Z Ran 1 test in 1.727s
2022-05-18T04:05:41.8011706Z 
2022-05-18T04:05:41.8011759Z OK
2022-05-18T04:05:41.8011872Z 
2022-05-18T04:05:41.8011973Z Generating XML reports...
2022-05-18T04:05:41.8045930Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040540.xml
2022-05-18T04:05:42.6321005Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw437lfi0
2022-05-18T04:05:42.6321469Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw437lfi0/_remote_module_non_scriptable.py
2022-05-18T04:05:42.8865689Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:05:42.8875804Z 
2022-05-18T04:05:42.8875949Z Running tests...
2022-05-18T04:05:42.8876400Z ----------------------------------------------------------------------
2022-05-18T04:05:43.2047024Z   test_remote_same_worker (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3412
2022-05-18T04:05:43.2069731Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3413
2022-05-18T04:05:43.2092935Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3414
2022-05-18T04:05:43.2117478Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3415
2022-05-18T04:05:43.8725648Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk25xrmbf
2022-05-18T04:05:43.8726394Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk25xrmbf/_remote_module_non_scriptable.py
2022-05-18T04:05:43.9224225Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp20lgs_2n
2022-05-18T04:05:43.9225000Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp20lgs_2n/_remote_module_non_scriptable.py
2022-05-18T04:05:43.9298003Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2rqei61m
2022-05-18T04:05:43.9299231Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2rqei61m/_remote_module_non_scriptable.py
2022-05-18T04:05:43.9308169Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4vxxc7ek
2022-05-18T04:05:43.9309996Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4vxxc7ek/_remote_module_non_scriptable.py
2022-05-18T04:05:44.1211084Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:05:44.1686308Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:05:44.1768932Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:05:44.1796917Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:05:44.7159533Z ok (1.828s)
2022-05-18T04:05:44.7159708Z 
2022-05-18T04:05:44.7160036Z ----------------------------------------------------------------------
2022-05-18T04:05:44.7160302Z Ran 1 test in 1.828s
2022-05-18T04:05:44.7160436Z 
2022-05-18T04:05:44.7160525Z OK
2022-05-18T04:05:44.7160927Z 
2022-05-18T04:05:44.7161030Z Generating XML reports...
2022-05-18T04:05:44.7195166Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040542.xml
2022-05-18T04:05:45.4884206Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpit6dyqfm
2022-05-18T04:05:45.4884919Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpit6dyqfm/_remote_module_non_scriptable.py
2022-05-18T04:05:45.7447019Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:05:45.7456765Z 
2022-05-18T04:05:45.7456842Z Running tests...
2022-05-18T04:05:45.7457576Z ----------------------------------------------------------------------
2022-05-18T04:05:46.0645579Z   test_remote_throw (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3631
2022-05-18T04:05:46.0669472Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3632
2022-05-18T04:05:46.0693206Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3633
2022-05-18T04:05:46.0717804Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3634
2022-05-18T04:05:46.7596313Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_g49r3n0
2022-05-18T04:05:46.7597403Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_g49r3n0/_remote_module_non_scriptable.py
2022-05-18T04:05:46.8057758Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphusqs1fl
2022-05-18T04:05:46.8058510Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphusqs1fl/_remote_module_non_scriptable.py
2022-05-18T04:05:46.8085501Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx226iesq
2022-05-18T04:05:46.8086852Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx226iesq/_remote_module_non_scriptable.py
2022-05-18T04:05:46.8811319Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz0en5lzb
2022-05-18T04:05:46.8812738Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz0en5lzb/_remote_module_non_scriptable.py
2022-05-18T04:05:47.0099664Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:05:47.0552823Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:05:47.0554953Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:05:47.1308272Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:05:47.3614338Z On WorkerInfo(id=1, name=worker1):
2022-05-18T04:05:47.3614789Z ValueError('Expected error')
2022-05-18T04:05:47.3614993Z Traceback (most recent call last):
2022-05-18T04:05:47.3615414Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function
2022-05-18T04:05:47.3615772Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-05-18T04:05:47.3616213Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 288, in raise_or_inc
2022-05-18T04:05:47.3616529Z     raise ValueError("Expected error")
2022-05-18T04:05:47.3616742Z ValueError: Expected error
2022-05-18T04:05:47.3616863Z 
2022-05-18T04:05:47.3768962Z On WorkerInfo(id=2, name=worker2):
2022-05-18T04:05:47.3769730Z ValueError('Expected error')
2022-05-18T04:05:47.3770260Z Traceback (most recent call last):
2022-05-18T04:05:47.3771155Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function
2022-05-18T04:05:47.3771882Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-05-18T04:05:47.3772863Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 288, in raise_or_inc
2022-05-18T04:05:47.3773539Z     raise ValueError("Expected error")
2022-05-18T04:05:47.3774333Z ValueError: Expected error
2022-05-18T04:05:47.3774578Z 
2022-05-18T04:05:47.3810808Z On WorkerInfo(id=0, name=worker0):
2022-05-18T04:05:47.3811573Z ValueError('Expected error')
2022-05-18T04:05:47.3811965Z Traceback (most recent call last):
2022-05-18T04:05:47.3812762Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function
2022-05-18T04:05:47.3813347Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-05-18T04:05:47.3814107Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 288, in raise_or_inc
2022-05-18T04:05:47.3814642Z     raise ValueError("Expected error")
2022-05-18T04:05:47.3814998Z ValueError: Expected error
2022-05-18T04:05:47.3815212Z 
2022-05-18T04:05:47.3875251Z On WorkerInfo(id=3, name=worker3):
2022-05-18T04:05:47.3875889Z ValueError('Expected error')
2022-05-18T04:05:47.3876329Z Traceback (most recent call last):
2022-05-18T04:05:47.3877216Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function
2022-05-18T04:05:47.3877934Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-05-18T04:05:47.3878900Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 288, in raise_or_inc
2022-05-18T04:05:47.3879544Z     raise ValueError("Expected error")
2022-05-18T04:05:47.3879994Z ValueError: Expected error
2022-05-18T04:05:47.3880269Z 
2022-05-18T04:05:47.5758410Z ok (1.830s)
2022-05-18T04:05:47.5758646Z 
2022-05-18T04:05:47.5758986Z ----------------------------------------------------------------------
2022-05-18T04:05:47.5759291Z Ran 1 test in 1.830s
2022-05-18T04:05:47.5759393Z 
2022-05-18T04:05:47.5759461Z OK
2022-05-18T04:05:47.5759554Z 
2022-05-18T04:05:47.5759647Z Generating XML reports...
2022-05-18T04:05:47.5794275Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040545.xml
2022-05-18T04:05:48.3571942Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxgdrgmgr
2022-05-18T04:05:48.3573499Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxgdrgmgr/_remote_module_non_scriptable.py
2022-05-18T04:05:48.6127082Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:05:48.6137433Z 
2022-05-18T04:05:48.6137755Z Running tests...
2022-05-18T04:05:48.6138409Z ----------------------------------------------------------------------
2022-05-18T04:05:48.9307295Z   test_remote_with_exception (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3850
2022-05-18T04:05:48.9330469Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3851
2022-05-18T04:05:48.9353836Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3852
2022-05-18T04:05:48.9379680Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3853
2022-05-18T04:05:49.5885144Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpceyud_vi
2022-05-18T04:05:49.5885853Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpceyud_vi/_remote_module_non_scriptable.py
2022-05-18T04:05:49.6449315Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdb5l3fl5
2022-05-18T04:05:49.6450204Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdb5l3fl5/_remote_module_non_scriptable.py
2022-05-18T04:05:49.6450900Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxe1f9et3
2022-05-18T04:05:49.6452018Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxe1f9et3/_remote_module_non_scriptable.py
2022-05-18T04:05:49.6714005Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg3ji2mkn
2022-05-18T04:05:49.6715011Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg3ji2mkn/_remote_module_non_scriptable.py
2022-05-18T04:05:49.8379275Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:05:49.8913430Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:05:49.8941278Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:05:49.9179873Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:05:50.1008994Z On WorkerInfo(id=1, name=worker1):
2022-05-18T04:05:50.1009636Z ValueError('Expected error')
2022-05-18T04:05:50.1010095Z Traceback (most recent call last):
2022-05-18T04:05:50.1010858Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function
2022-05-18T04:05:50.1011347Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-05-18T04:05:50.1011795Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 396, in raise_func
2022-05-18T04:05:50.1012107Z     raise ValueError(expected_err)
2022-05-18T04:05:50.1012319Z ValueError: Expected error
2022-05-18T04:05:50.1012447Z 
2022-05-18T04:05:50.1176501Z On WorkerInfo(id=0, name=worker0):
2022-05-18T04:05:50.1177800Z TypeError('no_result() takes 0 positional arguments but 1 was given')
2022-05-18T04:05:50.1178544Z Traceback (most recent call last):
2022-05-18T04:05:50.1179563Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function
2022-05-18T04:05:50.1180297Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-05-18T04:05:50.1180912Z TypeError: no_result() takes 0 positional arguments but 1 was given
2022-05-18T04:05:50.1181276Z 
2022-05-18T04:05:50.1181469Z On WorkerInfo(id=2, name=worker2):
2022-05-18T04:05:50.1181990Z ValueError('Expected error')
2022-05-18T04:05:50.1182429Z Traceback (most recent call last):
2022-05-18T04:05:50.1183511Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function
2022-05-18T04:05:50.1184453Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-05-18T04:05:50.1185286Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 396, in raise_func
2022-05-18T04:05:50.1185751Z     raise ValueError(expected_err)
2022-05-18T04:05:50.1186184Z ValueError: Expected error
2022-05-18T04:05:50.1186447Z 
2022-05-18T04:05:50.1205532Z On WorkerInfo(id=0, name=worker0):
2022-05-18T04:05:50.1206033Z ValueError('Expected error')
2022-05-18T04:05:50.1206357Z Traceback (most recent call last):
2022-05-18T04:05:50.1209381Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function
2022-05-18T04:05:50.1210299Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-05-18T04:05:50.1211365Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 396, in raise_func
2022-05-18T04:05:50.1211959Z     raise ValueError(expected_err)
2022-05-18T04:05:50.1212190Z ValueError: Expected error
2022-05-18T04:05:50.1212321Z 
2022-05-18T04:05:50.1589819Z On WorkerInfo(id=1, name=worker1):
2022-05-18T04:05:50.1590714Z TypeError('no_result() takes 0 positional arguments but 1 was given')
2022-05-18T04:05:50.1591266Z Traceback (most recent call last):
2022-05-18T04:05:50.1592120Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function
2022-05-18T04:05:50.1592839Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-05-18T04:05:50.1593439Z TypeError: no_result() takes 0 positional arguments but 1 was given
2022-05-18T04:05:50.1593786Z 
2022-05-18T04:05:50.1757082Z On WorkerInfo(id=3, name=worker3):
2022-05-18T04:05:50.1757717Z ValueError('Expected error')
2022-05-18T04:05:50.1758171Z Traceback (most recent call last):
2022-05-18T04:05:50.1760821Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function
2022-05-18T04:05:50.1761670Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-05-18T04:05:50.1762657Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 396, in raise_func
2022-05-18T04:05:50.1763308Z     raise ValueError(expected_err)
2022-05-18T04:05:50.1763758Z ValueError: Expected error
2022-05-18T04:05:50.1764022Z 
2022-05-18T04:05:50.1764214Z On WorkerInfo(id=3, name=worker3):
2022-05-18T04:05:50.1764896Z TypeError('no_result() takes 0 positional arguments but 1 was given')
2022-05-18T04:05:50.1766162Z Traceback (most recent call last):
2022-05-18T04:05:50.1766998Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function
2022-05-18T04:05:50.1767727Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-05-18T04:05:50.1768349Z TypeError: no_result() takes 0 positional arguments but 1 was given
2022-05-18T04:05:50.1768697Z 
2022-05-18T04:05:50.2136300Z On WorkerInfo(id=2, name=worker2):
2022-05-18T04:05:50.2138023Z TypeError('no_result() takes 0 positional arguments but 1 was given')
2022-05-18T04:05:50.2138550Z Traceback (most recent call last):
2022-05-18T04:05:50.2139283Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function
2022-05-18T04:05:50.2139904Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-05-18T04:05:50.2140530Z TypeError: no_result() takes 0 positional arguments but 1 was given
2022-05-18T04:05:50.2140892Z 
2022-05-18T04:05:50.4420047Z ok (1.828s)
2022-05-18T04:05:50.4420325Z 
2022-05-18T04:05:50.4420830Z ----------------------------------------------------------------------
2022-05-18T04:05:50.4421140Z Ran 1 test in 1.828s
2022-05-18T04:05:50.4421259Z 
2022-05-18T04:05:50.4421321Z OK
2022-05-18T04:05:50.4421414Z 
2022-05-18T04:05:50.4421523Z Generating XML reports...
2022-05-18T04:05:50.4455159Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040548.xml
2022-05-18T04:05:51.2149150Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7zol_il_
2022-05-18T04:05:51.2149882Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7zol_il_/_remote_module_non_scriptable.py
2022-05-18T04:05:51.4687550Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:05:51.4697189Z 
2022-05-18T04:05:51.4697324Z Running tests...
2022-05-18T04:05:51.4697898Z ----------------------------------------------------------------------
2022-05-18T04:05:51.7835128Z   test_return_future (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4069
2022-05-18T04:05:51.7858411Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4070
2022-05-18T04:05:51.7881906Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4071
2022-05-18T04:05:51.7905855Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4072
2022-05-18T04:05:52.3637506Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprzm7j3sw
2022-05-18T04:05:52.3638312Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprzm7j3sw/_remote_module_non_scriptable.py
2022-05-18T04:05:52.3702405Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3066s6py
2022-05-18T04:05:52.3703836Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3066s6py/_remote_module_non_scriptable.py
2022-05-18T04:05:52.4015810Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfocmdm1v
2022-05-18T04:05:52.4016543Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfocmdm1v/_remote_module_non_scriptable.py
2022-05-18T04:05:52.4056776Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcb4y1gmw
2022-05-18T04:05:52.4058724Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcb4y1gmw/_remote_module_non_scriptable.py
2022-05-18T04:05:52.6107006Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:05:52.6174291Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:05:52.6499776Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:05:52.6543765Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:05:53.0944032Z ok (1.624s)
2022-05-18T04:05:53.0945356Z 
2022-05-18T04:05:53.0945888Z ----------------------------------------------------------------------
2022-05-18T04:05:53.0946205Z Ran 1 test in 1.625s
2022-05-18T04:05:53.0946324Z 
2022-05-18T04:05:53.0946386Z OK
2022-05-18T04:05:53.0946481Z 
2022-05-18T04:05:53.0946590Z Generating XML reports...
2022-05-18T04:05:53.0979476Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040551.xml
2022-05-18T04:05:53.8599359Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuqv8tkvt
2022-05-18T04:05:53.8600299Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuqv8tkvt/_remote_module_non_scriptable.py
2022-05-18T04:05:54.1116263Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:05:54.1125736Z 
2022-05-18T04:05:54.1126123Z Running tests...
2022-05-18T04:05:54.1126528Z ----------------------------------------------------------------------
2022-05-18T04:05:54.4239148Z   test_return_future_async (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4288
2022-05-18T04:05:54.4262833Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4289
2022-05-18T04:05:54.4286281Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4290
2022-05-18T04:05:54.4310856Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4291
2022-05-18T04:05:55.0353809Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnh6q64rl
2022-05-18T04:05:55.0354759Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnh6q64rl/_remote_module_non_scriptable.py
2022-05-18T04:05:55.0457300Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp23i0ggbt
2022-05-18T04:05:55.0458112Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp23i0ggbt/_remote_module_non_scriptable.py
2022-05-18T04:05:55.0788342Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw1iqw3nu
2022-05-18T04:05:55.0789311Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw1iqw3nu/_remote_module_non_scriptable.py
2022-05-18T04:05:55.0869948Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdp968y3d
2022-05-18T04:05:55.0871171Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdp968y3d/_remote_module_non_scriptable.py
2022-05-18T04:05:55.2837878Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:05:55.2938356Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:05:55.3302635Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:05:55.3343834Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:05:55.8351582Z ok (1.722s)
2022-05-18T04:05:55.8351847Z 
2022-05-18T04:05:55.8352304Z ----------------------------------------------------------------------
2022-05-18T04:05:55.8352559Z Ran 1 test in 1.723s
2022-05-18T04:05:55.8352683Z 
2022-05-18T04:05:55.8352749Z OK
2022-05-18T04:05:55.8352843Z 
2022-05-18T04:05:55.8352937Z Generating XML reports...
2022-05-18T04:05:55.8388359Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040554.xml
2022-05-18T04:05:56.6078534Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6bfi47ap
2022-05-18T04:05:56.6079323Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6bfi47ap/_remote_module_non_scriptable.py
2022-05-18T04:05:56.8608745Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:05:56.8619160Z 
2022-05-18T04:05:56.8619681Z Running tests...
2022-05-18T04:05:56.8620089Z ----------------------------------------------------------------------
2022-05-18T04:05:57.1737155Z   test_return_future_remote (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4507
2022-05-18T04:05:57.1760391Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4508
2022-05-18T04:05:57.1784100Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4509
2022-05-18T04:05:57.1808615Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4510
2022-05-18T04:05:57.7782064Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp71j390ta
2022-05-18T04:05:57.7783240Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp71j390ta/_remote_module_non_scriptable.py
2022-05-18T04:05:57.8117629Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm4o4liz4
2022-05-18T04:05:57.8118360Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm4o4liz4/_remote_module_non_scriptable.py
2022-05-18T04:05:57.8416771Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkypyxj1i
2022-05-18T04:05:57.8417533Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkypyxj1i/_remote_module_non_scriptable.py
2022-05-18T04:05:57.8935273Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpiheszbed
2022-05-18T04:05:57.8936119Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpiheszbed/_remote_module_non_scriptable.py
2022-05-18T04:05:58.0252323Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:05:58.0587844Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:05:58.0856320Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:05:58.1415109Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:05:58.5848410Z ok (1.723s)
2022-05-18T04:05:58.5848576Z 
2022-05-18T04:05:58.5848948Z ----------------------------------------------------------------------
2022-05-18T04:05:58.5849227Z Ran 1 test in 1.723s
2022-05-18T04:05:58.5849349Z 
2022-05-18T04:05:58.5849413Z OK
2022-05-18T04:05:58.5849506Z 
2022-05-18T04:05:58.5849592Z Generating XML reports...
2022-05-18T04:05:58.5884706Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040556.xml
2022-05-18T04:05:59.3559379Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp74y1pleb
2022-05-18T04:05:59.3560137Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp74y1pleb/_remote_module_non_scriptable.py
2022-05-18T04:05:59.6081396Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:05:59.6091476Z 
2022-05-18T04:05:59.6091598Z Running tests...
2022-05-18T04:05:59.6091965Z ----------------------------------------------------------------------
2022-05-18T04:05:59.9243280Z   test_return_local_rrefs (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4726
2022-05-18T04:05:59.9267529Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4727
2022-05-18T04:05:59.9290528Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4728
2022-05-18T04:05:59.9315009Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4729
2022-05-18T04:06:00.5849880Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppcu3zdlv
2022-05-18T04:06:00.5850656Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppcu3zdlv/_remote_module_non_scriptable.py
2022-05-18T04:06:00.6258723Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvpqxhrkv
2022-05-18T04:06:00.6259895Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvpqxhrkv/_remote_module_non_scriptable.py
2022-05-18T04:06:00.6648419Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdbax8t_f
2022-05-18T04:06:00.6650181Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdbun595v
2022-05-18T04:06:00.6650939Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdbax8t_f/_remote_module_non_scriptable.py
2022-05-18T04:06:00.6651721Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdbun595v/_remote_module_non_scriptable.py
2022-05-18T04:06:00.8330007Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:06:00.8704778Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:06:00.9117361Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:06:00.9135678Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:06:01.4357707Z ok (1.826s)
2022-05-18T04:06:01.4357978Z 
2022-05-18T04:06:01.4358481Z ----------------------------------------------------------------------
2022-05-18T04:06:01.4358727Z Ran 1 test in 1.827s
2022-05-18T04:06:01.4358845Z 
2022-05-18T04:06:01.4358908Z OK
2022-05-18T04:06:01.4359002Z 
2022-05-18T04:06:01.4359093Z Generating XML reports...
2022-05-18T04:06:01.4394898Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040559.xml
2022-05-18T04:06:02.2077062Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph3l2geke
2022-05-18T04:06:02.2077962Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph3l2geke/_remote_module_non_scriptable.py
2022-05-18T04:06:02.4619857Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:06:02.4629793Z 
2022-05-18T04:06:02.4629885Z Running tests...
2022-05-18T04:06:02.4630970Z ----------------------------------------------------------------------
2022-05-18T04:06:02.7778990Z   test_rpc_barrier_all (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4945
2022-05-18T04:06:02.7802535Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4946
2022-05-18T04:06:02.7825787Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4947
2022-05-18T04:06:02.7850133Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4948
2022-05-18T04:06:03.3613314Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo11mzi7x
2022-05-18T04:06:03.3614078Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo11mzi7x/_remote_module_non_scriptable.py
2022-05-18T04:06:03.3638250Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpywpv4ahb
2022-05-18T04:06:03.3639762Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpywpv4ahb/_remote_module_non_scriptable.py
2022-05-18T04:06:03.4091559Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5ilgwdk5
2022-05-18T04:06:03.4092350Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5ilgwdk5/_remote_module_non_scriptable.py
2022-05-18T04:06:03.4232832Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpveow6_vm
2022-05-18T04:06:03.4233874Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpveow6_vm/_remote_module_non_scriptable.py
2022-05-18T04:06:03.6131125Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:06:03.6148196Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:06:03.6585217Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:06:03.6729204Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:06:04.0889607Z ok (1.626s)
2022-05-18T04:06:04.0889867Z 
2022-05-18T04:06:04.0890186Z ----------------------------------------------------------------------
2022-05-18T04:06:04.0890420Z Ran 1 test in 1.626s
2022-05-18T04:06:04.0890545Z 
2022-05-18T04:06:04.0890607Z OK
2022-05-18T04:06:04.0890699Z 
2022-05-18T04:06:04.0890794Z Generating XML reports...
2022-05-18T04:06:04.0925796Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040602.xml
2022-05-18T04:06:04.8621064Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwtk3tycm
2022-05-18T04:06:04.8621708Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwtk3tycm/_remote_module_non_scriptable.py
2022-05-18T04:06:05.1154399Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:06:05.1163661Z 
2022-05-18T04:06:05.1163745Z Running tests...
2022-05-18T04:06:05.1164877Z ----------------------------------------------------------------------
2022-05-18T04:06:05.4331915Z   test_rpc_barrier_multithreaded (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5164
2022-05-18T04:06:05.4354553Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5165
2022-05-18T04:06:05.4378069Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5166
2022-05-18T04:06:05.4402845Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5167
2022-05-18T04:06:06.0478208Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp002vvudj
2022-05-18T04:06:06.0479540Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp002vvudj/_remote_module_non_scriptable.py
2022-05-18T04:06:06.0582300Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpws8nh139
2022-05-18T04:06:06.0583988Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpws8nh139/_remote_module_non_scriptable.py
2022-05-18T04:06:06.0777976Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkijhesso
2022-05-18T04:06:06.0779994Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkijhesso/_remote_module_non_scriptable.py
2022-05-18T04:06:06.0942810Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk5qtl52_
2022-05-18T04:06:06.0943454Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk5qtl52_/_remote_module_non_scriptable.py
2022-05-18T04:06:06.2959143Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:06:06.3279343Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:06:06.3412557Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:06:06.3474792Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:06:06.7440517Z ok (1.627s)
2022-05-18T04:06:06.7440746Z 
2022-05-18T04:06:06.7441231Z ----------------------------------------------------------------------
2022-05-18T04:06:06.7441706Z Ran 1 test in 1.628s
2022-05-18T04:06:06.7441912Z 
2022-05-18T04:06:06.7442005Z OK
2022-05-18T04:06:06.7442104Z 
2022-05-18T04:06:06.7442198Z Generating XML reports...
2022-05-18T04:06:06.7477163Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040605.xml
2022-05-18T04:06:07.5267824Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8itvkbmo
2022-05-18T04:06:07.5268784Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8itvkbmo/_remote_module_non_scriptable.py
2022-05-18T04:06:07.7808303Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:06:07.7822495Z 
2022-05-18T04:06:07.7822829Z Running tests...
2022-05-18T04:06:07.7823437Z ----------------------------------------------------------------------
2022-05-18T04:06:08.1012104Z   test_rpc_barrier_partial_subset (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5395
2022-05-18T04:06:08.1035479Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5396
2022-05-18T04:06:08.1058554Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5397
2022-05-18T04:06:08.1082537Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5398
2022-05-18T04:06:08.6783425Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfcr_ccx2
2022-05-18T04:06:08.6784235Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfcr_ccx2/_remote_module_non_scriptable.py
2022-05-18T04:06:08.6940566Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmr4iz6ql
2022-05-18T04:06:08.6941587Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmr4iz6ql/_remote_module_non_scriptable.py
2022-05-18T04:06:08.7196158Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphueti8ou
2022-05-18T04:06:08.7197067Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphueti8ou/_remote_module_non_scriptable.py
2022-05-18T04:06:08.7404205Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt489qede
2022-05-18T04:06:08.7405158Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt489qede/_remote_module_non_scriptable.py
2022-05-18T04:06:08.9232837Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:06:08.9423960Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:06:08.9696323Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:06:08.9869907Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:06:09.4120168Z ok (1.629s)
2022-05-18T04:06:09.4120738Z 
2022-05-18T04:06:09.4121505Z ----------------------------------------------------------------------
2022-05-18T04:06:09.4121949Z Ran 1 test in 1.630s
2022-05-18T04:06:09.4122134Z 
2022-05-18T04:06:09.4122238Z OK
2022-05-18T04:06:09.4122391Z 
2022-05-18T04:06:09.4122546Z Generating XML reports...
2022-05-18T04:06:09.4157945Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040607.xml
2022-05-18T04:06:10.1856081Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxx87qhf7
2022-05-18T04:06:10.1856966Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxx87qhf7/_remote_module_non_scriptable.py
2022-05-18T04:06:10.4391410Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:06:10.4400507Z 
2022-05-18T04:06:10.4400648Z Running tests...
2022-05-18T04:06:10.4401070Z ----------------------------------------------------------------------
2022-05-18T04:06:10.7531087Z   test_rpc_barrier_subset (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5614
2022-05-18T04:06:10.7554570Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5615
2022-05-18T04:06:10.7577701Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5616
2022-05-18T04:06:10.7601737Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5617
2022-05-18T04:06:11.4177628Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3dxda_76
2022-05-18T04:06:11.4178430Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3dxda_76/_remote_module_non_scriptable.py
2022-05-18T04:06:11.4538098Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgn9yas6l
2022-05-18T04:06:11.4538885Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgn9yas6l/_remote_module_non_scriptable.py
2022-05-18T04:06:11.4682742Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt2tgp8o8
2022-05-18T04:06:11.4684702Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt2tgp8o8/_remote_module_non_scriptable.py
2022-05-18T04:06:11.4842491Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppfdxw803
2022-05-18T04:06:11.4843633Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppfdxw803/_remote_module_non_scriptable.py
2022-05-18T04:06:11.6670057Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:06:11.7004638Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:06:11.7164102Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:06:11.7313572Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:06:12.2644908Z ok (1.824s)
2022-05-18T04:06:12.2645160Z 
2022-05-18T04:06:12.2645694Z ----------------------------------------------------------------------
2022-05-18T04:06:12.2645981Z Ran 1 test in 1.824s
2022-05-18T04:06:12.2646097Z 
2022-05-18T04:06:12.2646147Z OK
2022-05-18T04:06:12.2646243Z 
2022-05-18T04:06:12.2646339Z Generating XML reports...
2022-05-18T04:06:12.2681269Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040610.xml
2022-05-18T04:06:13.0511010Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbba15h9m
2022-05-18T04:06:13.0511870Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbba15h9m/_remote_module_non_scriptable.py
2022-05-18T04:06:13.3047828Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:06:13.3057273Z 
2022-05-18T04:06:13.3057408Z Running tests...
2022-05-18T04:06:13.3057829Z ----------------------------------------------------------------------
2022-05-18T04:06:13.6187215Z   test_rpc_profiling_async_function (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5833
2022-05-18T04:06:13.6210069Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5834
2022-05-18T04:06:13.6233514Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5835
2022-05-18T04:06:13.6257609Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5836
2022-05-18T04:06:14.2225229Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2k7v8cwm
2022-05-18T04:06:14.2226138Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2k7v8cwm/_remote_module_non_scriptable.py
2022-05-18T04:06:14.2451694Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9w47zxcw
2022-05-18T04:06:14.2453190Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9w47zxcw/_remote_module_non_scriptable.py
2022-05-18T04:06:14.2685424Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6xh6c45a
2022-05-18T04:06:14.2686868Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6xh6c45a/_remote_module_non_scriptable.py
2022-05-18T04:06:14.2917136Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnoood28p
2022-05-18T04:06:14.2918289Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnoood28p/_remote_module_non_scriptable.py
2022-05-18T04:06:14.4701415Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:06:14.4972139Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:06:14.5164320Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:06:14.5422639Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:06:14.7873112Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:06:14.7972162Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:06:14.8074516Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:06:14.8075215Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T04:06:14.8077315Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:06:14.8078288Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:06:14.8079014Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:06:14.8079905Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:06:16.1315316Z ok (2.826s)
2022-05-18T04:06:16.1315586Z 
2022-05-18T04:06:16.1316126Z ----------------------------------------------------------------------
2022-05-18T04:06:16.1316431Z Ran 1 test in 2.826s
2022-05-18T04:06:16.1316545Z 
2022-05-18T04:06:16.1316594Z OK
2022-05-18T04:06:16.1316708Z 
2022-05-18T04:06:16.1316799Z Generating XML reports...
2022-05-18T04:06:16.1351141Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040613.xml
2022-05-18T04:06:16.9069840Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk4o7t8i0
2022-05-18T04:06:17.1592570Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk4o7t8i0/_remote_module_non_scriptable.py
2022-05-18T04:06:17.1593196Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:06:17.1602858Z 
2022-05-18T04:06:17.1602985Z Running tests...
2022-05-18T04:06:17.1603309Z ----------------------------------------------------------------------
2022-05-18T04:06:17.4768103Z   test_rpc_profiling_async_function_single_threaded (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6064
2022-05-18T04:06:17.4789816Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6065
2022-05-18T04:06:17.4813207Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6066
2022-05-18T04:06:17.4837969Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6067
2022-05-18T04:06:18.0816108Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsvqk0yfr
2022-05-18T04:06:18.0816864Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsvqk0yfr/_remote_module_non_scriptable.py
2022-05-18T04:06:18.0872635Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw4i9_wts
2022-05-18T04:06:18.0874024Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw4i9_wts/_remote_module_non_scriptable.py
2022-05-18T04:06:18.1214629Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa22bq4mh
2022-05-18T04:06:18.1215446Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa22bq4mh/_remote_module_non_scriptable.py
2022-05-18T04:06:18.1241856Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpiuiv6k61
2022-05-18T04:06:18.1243520Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpiuiv6k61/_remote_module_non_scriptable.py
2022-05-18T04:06:18.3322715Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:06:18.3337124Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:06:18.3693946Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:06:18.3722480Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:06:18.6433643Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:06:18.6534310Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:06:18.6639211Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T04:06:18.6640662Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:06:18.6642556Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:06:18.6643691Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:06:18.6644662Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:06:18.6645465Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:06:19.9895710Z ok (2.829s)
2022-05-18T04:06:19.9895965Z 
2022-05-18T04:06:19.9896486Z ----------------------------------------------------------------------
2022-05-18T04:06:19.9896835Z Ran 1 test in 2.829s
2022-05-18T04:06:19.9896948Z 
2022-05-18T04:06:19.9897009Z OK
2022-05-18T04:06:19.9897087Z 
2022-05-18T04:06:19.9897183Z Generating XML reports...
2022-05-18T04:06:19.9931090Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040617.xml
2022-05-18T04:06:20.7670914Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbsxlpx46
2022-05-18T04:06:20.7671639Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbsxlpx46/_remote_module_non_scriptable.py
2022-05-18T04:06:21.0198858Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:06:21.0208721Z 
2022-05-18T04:06:21.0209061Z Running tests...
2022-05-18T04:06:21.0209778Z ----------------------------------------------------------------------
2022-05-18T04:06:21.3357913Z   test_rpc_profiling_remote_record_function (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6295
2022-05-18T04:06:21.3379640Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6296
2022-05-18T04:06:21.3403349Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6297
2022-05-18T04:06:21.3427573Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6298
2022-05-18T04:06:21.9065495Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpukowmzaf
2022-05-18T04:06:21.9066285Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpukowmzaf/_remote_module_non_scriptable.py
2022-05-18T04:06:21.9091149Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnmk458ls
2022-05-18T04:06:21.9092296Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnmk458ls/_remote_module_non_scriptable.py
2022-05-18T04:06:21.9249649Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcs1txw6y
2022-05-18T04:06:21.9251075Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcs1txw6y/_remote_module_non_scriptable.py
2022-05-18T04:06:21.9732290Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmf8pv2z9
2022-05-18T04:06:21.9733560Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmf8pv2z9/_remote_module_non_scriptable.py
2022-05-18T04:06:22.1553991Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:06:22.1566137Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:06:22.1741481Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:06:22.2192334Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:06:22.7467834Z ok (1.726s)
2022-05-18T04:06:22.7467983Z 
2022-05-18T04:06:22.7468438Z ----------------------------------------------------------------------
2022-05-18T04:06:22.7468912Z Ran 1 test in 1.726s
2022-05-18T04:06:22.7469119Z 
2022-05-18T04:06:22.7469216Z OK
2022-05-18T04:06:22.7469310Z 
2022-05-18T04:06:22.7469393Z Generating XML reports...
2022-05-18T04:06:22.7502384Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040621.xml
2022-05-18T04:06:23.5179569Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1kqey_ph
2022-05-18T04:06:23.5180274Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1kqey_ph/_remote_module_non_scriptable.py
2022-05-18T04:06:23.7707622Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:06:23.7717224Z 
2022-05-18T04:06:23.7717323Z Running tests...
2022-05-18T04:06:23.7717908Z ----------------------------------------------------------------------
2022-05-18T04:06:24.0860142Z   test_rpc_return_rref (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6514
2022-05-18T04:06:24.0882906Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6515
2022-05-18T04:06:24.0907407Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6516
2022-05-18T04:06:24.0931775Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6517
2022-05-18T04:06:24.7835657Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdck_1_eh
2022-05-18T04:06:24.7837057Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdck_1_eh/_remote_module_non_scriptable.py
2022-05-18T04:06:24.8261385Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpph5ymdae
2022-05-18T04:06:24.8262108Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpph5ymdae/_remote_module_non_scriptable.py
2022-05-18T04:06:24.8346827Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprgx86_8p
2022-05-18T04:06:24.8348518Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprgx86_8p/_remote_module_non_scriptable.py
2022-05-18T04:06:24.8386697Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1y44jfm1
2022-05-18T04:06:24.8388445Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1y44jfm1/_remote_module_non_scriptable.py
2022-05-18T04:06:25.0321216Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:06:25.0721109Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:06:25.0822198Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:06:25.0839338Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:06:25.5974402Z ok (1.825s)
2022-05-18T04:06:25.5974677Z 
2022-05-18T04:06:25.5975206Z ----------------------------------------------------------------------
2022-05-18T04:06:25.5975803Z Ran 1 test in 1.826s
2022-05-18T04:06:25.5975919Z 
2022-05-18T04:06:25.5975983Z OK
2022-05-18T04:06:25.5976076Z 
2022-05-18T04:06:25.5976170Z Generating XML reports...
2022-05-18T04:06:25.6010168Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040623.xml
2022-05-18T04:06:26.3677383Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyeh13xa_
2022-05-18T04:06:26.3677967Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyeh13xa_/_remote_module_non_scriptable.py
2022-05-18T04:06:26.6207980Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:06:26.6218633Z 
2022-05-18T04:06:26.6219033Z Running tests...
2022-05-18T04:06:26.6219417Z ----------------------------------------------------------------------
2022-05-18T04:06:26.9351021Z   test_rpc_timeouts (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6733
2022-05-18T04:06:26.9373519Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6734
2022-05-18T04:06:26.9396765Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6735
2022-05-18T04:06:26.9420862Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6736
2022-05-18T04:06:27.5535112Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp28f2gcqs
2022-05-18T04:06:27.5536268Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpitiq03xt
2022-05-18T04:06:27.5537249Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp28f2gcqs/_remote_module_non_scriptable.py
2022-05-18T04:06:27.5538156Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpitiq03xt/_remote_module_non_scriptable.py
2022-05-18T04:06:27.5633536Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpddneg3hy
2022-05-18T04:06:27.5635120Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpddneg3hy/_remote_module_non_scriptable.py
2022-05-18T04:06:27.5749448Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpafz83stm
2022-05-18T04:06:27.5750737Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpafz83stm/_remote_module_non_scriptable.py
2022-05-18T04:06:27.7993452Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:06:27.8016323Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:06:27.8111399Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:06:27.8213383Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:06:34.4555046Z ok (7.833s)
2022-05-18T04:06:34.4555336Z 
2022-05-18T04:06:34.4555812Z ----------------------------------------------------------------------
2022-05-18T04:06:34.4556290Z Ran 1 test in 7.834s
2022-05-18T04:06:34.4556496Z 
2022-05-18T04:06:34.4556609Z OK
2022-05-18T04:06:34.4556760Z 
2022-05-18T04:06:34.4556925Z Generating XML reports...
2022-05-18T04:06:34.4590472Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040626.xml
2022-05-18T04:06:35.2223395Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1z7ie8fl
2022-05-18T04:06:35.2224870Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1z7ie8fl/_remote_module_non_scriptable.py
2022-05-18T04:06:35.4760687Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:06:35.4770109Z 
2022-05-18T04:06:35.4770184Z Running tests...
2022-05-18T04:06:35.4770893Z ----------------------------------------------------------------------
2022-05-18T04:06:35.8025084Z   test_rref_context_debug_info (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6952
2022-05-18T04:06:35.8048070Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6953
2022-05-18T04:06:35.8071637Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6954
2022-05-18T04:06:35.8095695Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6955
2022-05-18T04:06:36.4709680Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuacksw_z
2022-05-18T04:06:36.4710417Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuacksw_z/_remote_module_non_scriptable.py
2022-05-18T04:06:36.5168049Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpaq6aopdm
2022-05-18T04:06:36.5168837Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpaq6aopdm/_remote_module_non_scriptable.py
2022-05-18T04:06:36.5433028Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg25h_2cx
2022-05-18T04:06:36.5433760Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg25h_2cx/_remote_module_non_scriptable.py
2022-05-18T04:06:36.5645724Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprvm_nnwy
2022-05-18T04:06:36.5646942Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprvm_nnwy/_remote_module_non_scriptable.py
2022-05-18T04:06:36.7211909Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:06:36.7665611Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:06:36.7936048Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:06:36.8140643Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:06:37.0613129Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:06:37.0712986Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:06:37.0813575Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:06:37.0821518Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T04:06:37.0822836Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:06:37.0824176Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:06:37.0825999Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:06:37.0827572Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:06:37.5141480Z ok (2.037s)
2022-05-18T04:06:37.5141670Z 
2022-05-18T04:06:37.5142015Z ----------------------------------------------------------------------
2022-05-18T04:06:37.5142265Z Ran 1 test in 2.037s
2022-05-18T04:06:37.5142391Z 
2022-05-18T04:06:37.5142453Z OK
2022-05-18T04:06:37.5142533Z 
2022-05-18T04:06:37.5142635Z Generating XML reports...
2022-05-18T04:06:37.5174957Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040635.xml
2022-05-18T04:06:38.2820363Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp48zadbdm
2022-05-18T04:06:38.2821351Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp48zadbdm/_remote_module_non_scriptable.py
2022-05-18T04:06:38.5335094Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:06:38.5344965Z 
2022-05-18T04:06:38.5345097Z Running tests...
2022-05-18T04:06:38.5345561Z ----------------------------------------------------------------------
2022-05-18T04:06:38.8501012Z   test_rref_forward_chain (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7184
2022-05-18T04:06:38.8523296Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7185
2022-05-18T04:06:38.8546063Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7186
2022-05-18T04:06:38.8570681Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7187
2022-05-18T04:06:39.5072835Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqixpn01v
2022-05-18T04:06:39.5073638Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqixpn01v/_remote_module_non_scriptable.py
2022-05-18T04:06:39.5284603Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb221r7qo
2022-05-18T04:06:39.5286405Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb221r7qo/_remote_module_non_scriptable.py
2022-05-18T04:06:39.5824135Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv8nab_9k
2022-05-18T04:06:39.5824951Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv8nab_9k/_remote_module_non_scriptable.py
2022-05-18T04:06:39.6029698Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpobm7x4ec
2022-05-18T04:06:39.6030849Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpobm7x4ec/_remote_module_non_scriptable.py
2022-05-18T04:06:39.7556469Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:06:39.7757719Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:06:39.8287214Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:06:39.8510071Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:06:40.3611766Z ok (1.826s)
2022-05-18T04:06:40.3611932Z 
2022-05-18T04:06:40.3614031Z ----------------------------------------------------------------------
2022-05-18T04:06:40.3614550Z Ran 1 test in 1.827s
2022-05-18T04:06:40.3614733Z 
2022-05-18T04:06:40.3614784Z OK
2022-05-18T04:06:40.3614878Z 
2022-05-18T04:06:40.3614982Z Generating XML reports...
2022-05-18T04:06:40.3647608Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040638.xml
2022-05-18T04:06:41.1344752Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2ubz1__d
2022-05-18T04:06:41.1345606Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2ubz1__d/_remote_module_non_scriptable.py
2022-05-18T04:06:41.3863396Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:06:41.3873434Z 
2022-05-18T04:06:41.3873750Z Running tests...
2022-05-18T04:06:41.3874404Z ----------------------------------------------------------------------
2022-05-18T04:06:41.7081545Z   test_rref_get_future (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7403
2022-05-18T04:06:41.7104701Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7404
2022-05-18T04:06:41.7128218Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7405
2022-05-18T04:06:41.7152848Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7406
2022-05-18T04:06:42.3486626Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd5wgnj_u
2022-05-18T04:06:42.3487365Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd5wgnj_u/_remote_module_non_scriptable.py
2022-05-18T04:06:42.3964413Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0__9s46n
2022-05-18T04:06:42.3965151Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzl0m1o1i
2022-05-18T04:06:42.3965789Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0__9s46n/_remote_module_non_scriptable.py
2022-05-18T04:06:42.3966660Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzl0m1o1i/_remote_module_non_scriptable.py
2022-05-18T04:06:42.4023211Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbsrkqr_2
2022-05-18T04:06:42.4024544Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbsrkqr_2/_remote_module_non_scriptable.py
2022-05-18T04:06:42.5977665Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:06:42.6425723Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:06:42.6436572Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:06:42.6511407Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:06:43.1193312Z ok (1.732s)
2022-05-18T04:06:43.1193520Z 
2022-05-18T04:06:43.1193948Z ----------------------------------------------------------------------
2022-05-18T04:06:43.1194382Z Ran 1 test in 1.732s
2022-05-18T04:06:43.1194560Z 
2022-05-18T04:06:43.1194654Z OK
2022-05-18T04:06:43.1194791Z 
2022-05-18T04:06:43.1194938Z Generating XML reports...
2022-05-18T04:06:43.1229076Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040641.xml
2022-05-18T04:06:43.8966326Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo64esckh
2022-05-18T04:06:43.8967071Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo64esckh/_remote_module_non_scriptable.py
2022-05-18T04:06:44.1506912Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:06:44.1516104Z 
2022-05-18T04:06:44.1516202Z Running tests...
2022-05-18T04:06:44.1516791Z ----------------------------------------------------------------------
2022-05-18T04:06:44.4704459Z   test_rref_leak (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7622
2022-05-18T04:06:44.4727648Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7623
2022-05-18T04:06:44.4752172Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7624
2022-05-18T04:06:44.4776060Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7625
2022-05-18T04:06:45.1494649Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyrfm1fzh
2022-05-18T04:06:45.1495652Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyrfm1fzh/_remote_module_non_scriptable.py
2022-05-18T04:06:45.1566165Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj6uvzis5
2022-05-18T04:06:45.1567538Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj6uvzis5/_remote_module_non_scriptable.py
2022-05-18T04:06:45.1648305Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptddzanri
2022-05-18T04:06:45.1649977Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptddzanri/_remote_module_non_scriptable.py
2022-05-18T04:06:45.1676213Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp81p8x4wl
2022-05-18T04:06:45.1677732Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp81p8x4wl/_remote_module_non_scriptable.py
2022-05-18T04:06:45.3989506Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:06:45.4037491Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:06:45.4118896Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:06:45.4156217Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:06:45.6531769Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:06:45.6734995Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:06:45.6736069Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:06:45.6736864Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T04:06:45.6738163Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:06:45.6739338Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:06:45.6740491Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:06:45.6741628Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:06:45.7140560Z [W rref_context.cpp:156] Detected RRef Leaks during shutdown. This usually occurs when the application code still holds references to RRef instances when calling shutdown(). If the program has completed correctly and the process is exiting, it is OK to ignore these leaks. However, if you program will keep running after this, these leaks could result in memory leaks on RRef owners. Please make sure all RRefs are out of scope and Python GC has deleted them before calling shutdown(): 
2022-05-18T04:06:45.7142381Z Leaking RRef GloballyUniqueId(created_on=1, local_id=0) with fork Id GloballyUniqueId(created_on=1, local_id=1)
2022-05-18T04:06:45.7142856Z 
2022-05-18T04:06:45.7148525Z [W rref_context.cpp:156] Detected RRef Leaks during shutdown. This usually occurs when the application code still holds references to RRef instances when calling shutdown(). If the program has completed correctly and the process is exiting, it is OK to ignore these leaks. However, if you program will keep running after this, these leaks could result in memory leaks on RRef owners. Please make sure all RRefs are out of scope and Python GC has deleted them before calling shutdown(): 
2022-05-18T04:06:45.7150184Z Leaking RRef GloballyUniqueId(created_on=3, local_id=0) with fork Id GloballyUniqueId(created_on=3, local_id=1)
2022-05-18T04:06:45.7150624Z 
2022-05-18T04:06:45.7173718Z [W rref_context.cpp:156] Detected RRef Leaks during shutdown. This usually occurs when the application code still holds references to RRef instances when calling shutdown(). If the program has completed correctly and the process is exiting, it is OK to ignore these leaks. However, if you program will keep running after this, these leaks could result in memory leaks on RRef owners. Please make sure all RRefs are out of scope and Python GC has deleted them before calling shutdown(): 
2022-05-18T04:06:45.7175853Z Leaking RRef GloballyUniqueId(created_on=0, local_id=0) with fork Id GloballyUniqueId(created_on=0, local_id=1)
2022-05-18T04:06:45.7176334Z 
2022-05-18T04:06:45.7220362Z [W rref_context.cpp:156] Detected RRef Leaks during shutdown. This usually occurs when the application code still holds references to RRef instances when calling shutdown(). If the program has completed correctly and the process is exiting, it is OK to ignore these leaks. However, if you program will keep running after this, these leaks could result in memory leaks on RRef owners. Please make sure all RRefs are out of scope and Python GC has deleted them before calling shutdown(): 
2022-05-18T04:06:45.7221739Z Leaking RRef GloballyUniqueId(created_on=2, local_id=0) with fork Id GloballyUniqueId(created_on=2, local_id=1)
2022-05-18T04:06:45.7222623Z 
2022-05-18T04:06:45.9819102Z ok (1.830s)
2022-05-18T04:06:45.9819383Z 
2022-05-18T04:06:45.9819825Z ----------------------------------------------------------------------
2022-05-18T04:06:45.9820075Z Ran 1 test in 1.830s
2022-05-18T04:06:45.9820180Z 
2022-05-18T04:06:45.9820244Z OK
2022-05-18T04:06:45.9820336Z 
2022-05-18T04:06:45.9820428Z Generating XML reports...
2022-05-18T04:06:45.9854337Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040644.xml
2022-05-18T04:06:46.7575526Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpci92ztgj
2022-05-18T04:06:46.7576090Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpci92ztgj/_remote_module_non_scriptable.py
2022-05-18T04:06:47.0108882Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:06:47.0118415Z 
2022-05-18T04:06:47.0118549Z Running tests...
2022-05-18T04:06:47.0118990Z ----------------------------------------------------------------------
2022-05-18T04:06:47.3308276Z   test_rref_proxy_class (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7853
2022-05-18T04:06:47.3330942Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7854
2022-05-18T04:06:47.3354486Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7855
2022-05-18T04:06:47.3379802Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7856
2022-05-18T04:06:47.9095706Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1she8jmk
2022-05-18T04:06:47.9096502Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1she8jmk/_remote_module_non_scriptable.py
2022-05-18T04:06:47.9169436Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkkisxpln
2022-05-18T04:06:47.9170957Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkkisxpln/_remote_module_non_scriptable.py
2022-05-18T04:06:47.9205361Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpoaz_g8w2
2022-05-18T04:06:47.9207309Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpoaz_g8w2/_remote_module_non_scriptable.py
2022-05-18T04:06:47.9637378Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp90b4epcc
2022-05-18T04:06:47.9638424Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp90b4epcc/_remote_module_non_scriptable.py
2022-05-18T04:06:48.1587489Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:06:48.1645891Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:06:48.1668197Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:06:48.2108884Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:06:48.7419117Z ok (1.730s)
2022-05-18T04:06:48.7419416Z 
2022-05-18T04:06:48.7419802Z ----------------------------------------------------------------------
2022-05-18T04:06:48.7420251Z Ran 1 test in 1.730s
2022-05-18T04:06:48.7420465Z 
2022-05-18T04:06:48.7420584Z OK
2022-05-18T04:06:48.7420719Z 
2022-05-18T04:06:48.7420833Z Generating XML reports...
2022-05-18T04:06:48.7453882Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040647.xml
2022-05-18T04:06:49.5109137Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpluld6h1h
2022-05-18T04:06:49.5109903Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpluld6h1h/_remote_module_non_scriptable.py
2022-05-18T04:06:49.7647446Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:06:49.7657513Z 
2022-05-18T04:06:49.7657900Z Running tests...
2022-05-18T04:06:49.7658466Z ----------------------------------------------------------------------
2022-05-18T04:06:50.0828595Z   test_rref_proxy_class_self (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8072
2022-05-18T04:06:50.0853069Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8073
2022-05-18T04:06:50.0876071Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8074
2022-05-18T04:06:50.0900750Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8075
2022-05-18T04:06:50.6911475Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_3tdfi89
2022-05-18T04:06:50.6912229Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_3tdfi89/_remote_module_non_scriptable.py
2022-05-18T04:06:50.7242234Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpavpt8hfc
2022-05-18T04:06:50.7243487Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpavpt8hfc/_remote_module_non_scriptable.py
2022-05-18T04:06:50.7310300Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptyljk_cj
2022-05-18T04:06:50.7311662Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptyljk_cj/_remote_module_non_scriptable.py
2022-05-18T04:06:50.7377502Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqx_s9fie
2022-05-18T04:06:50.7379136Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqx_s9fie/_remote_module_non_scriptable.py
2022-05-18T04:06:50.9410365Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:06:50.9716079Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:06:50.9781093Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:06:50.9836939Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:06:51.3940851Z ok (1.628s)
2022-05-18T04:06:51.3941059Z 
2022-05-18T04:06:51.3941410Z ----------------------------------------------------------------------
2022-05-18T04:06:51.3941659Z Ran 1 test in 1.628s
2022-05-18T04:06:51.3941788Z 
2022-05-18T04:06:51.3941851Z OK
2022-05-18T04:06:51.3941980Z 
2022-05-18T04:06:51.3942076Z Generating XML reports...
2022-05-18T04:06:51.3975775Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040649.xml
2022-05-18T04:06:52.1762847Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppljgwgti
2022-05-18T04:06:52.1763746Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppljgwgti/_remote_module_non_scriptable.py
2022-05-18T04:06:52.4345582Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:06:52.4355248Z 
2022-05-18T04:06:52.4355370Z Running tests...
2022-05-18T04:06:52.4355816Z ----------------------------------------------------------------------
2022-05-18T04:06:52.7737447Z   test_rref_proxy_non_exist (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8291
2022-05-18T04:06:52.7760245Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8292
2022-05-18T04:06:52.7784203Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8293
2022-05-18T04:06:52.7807637Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8294
2022-05-18T04:06:53.3941579Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9s9eb1xd
2022-05-18T04:06:53.3942336Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9s9eb1xd/_remote_module_non_scriptable.py
2022-05-18T04:06:53.4247510Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpakwan326
2022-05-18T04:06:53.4248339Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpakwan326/_remote_module_non_scriptable.py
2022-05-18T04:06:53.4299644Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_a2ny18j
2022-05-18T04:06:53.4300457Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_a2ny18j/_remote_module_non_scriptable.py
2022-05-18T04:06:53.4432832Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxwpwxz0p
2022-05-18T04:06:53.4433969Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxwpwxz0p/_remote_module_non_scriptable.py
2022-05-18T04:06:53.6427412Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:06:53.6730951Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:06:53.6781062Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:06:53.6910415Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:06:54.1848356Z ok (1.749s)
2022-05-18T04:06:54.1848742Z 
2022-05-18T04:06:54.1849745Z ----------------------------------------------------------------------
2022-05-18T04:06:54.1850155Z Ran 1 test in 1.749s
2022-05-18T04:06:54.1850278Z 
2022-05-18T04:06:54.1850340Z OK
2022-05-18T04:06:54.1850441Z 
2022-05-18T04:06:54.1850535Z Generating XML reports...
2022-05-18T04:06:54.1882958Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040652.xml
2022-05-18T04:06:54.9580359Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp24r8q2ul
2022-05-18T04:06:54.9581267Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp24r8q2ul/_remote_module_non_scriptable.py
2022-05-18T04:06:55.2139219Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:06:55.2149521Z 
2022-05-18T04:06:55.2149960Z Running tests...
2022-05-18T04:06:55.2150370Z ----------------------------------------------------------------------
2022-05-18T04:06:55.5380236Z   test_rref_proxy_reuse (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8510
2022-05-18T04:06:55.5403547Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8511
2022-05-18T04:06:55.5427767Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8512
2022-05-18T04:06:55.5452784Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8513
2022-05-18T04:06:56.1775916Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphb0iq3sp
2022-05-18T04:06:56.1776709Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphb0iq3sp/_remote_module_non_scriptable.py
2022-05-18T04:06:56.1871683Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf55vgl3i
2022-05-18T04:06:56.1872794Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf55vgl3i/_remote_module_non_scriptable.py
2022-05-18T04:06:56.1952569Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf8lomrnk
2022-05-18T04:06:56.1953820Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf8lomrnk/_remote_module_non_scriptable.py
2022-05-18T04:06:56.2130795Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp82i0vcxi
2022-05-18T04:06:56.2131939Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp82i0vcxi/_remote_module_non_scriptable.py
2022-05-18T04:06:56.4285724Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:06:56.4348432Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:06:56.4433382Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:06:56.4641876Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:06:56.9494173Z ok (1.734s)
2022-05-18T04:06:56.9494401Z 
2022-05-18T04:06:56.9494921Z ----------------------------------------------------------------------
2022-05-18T04:06:56.9495406Z Ran 1 test in 1.734s
2022-05-18T04:06:56.9495604Z 
2022-05-18T04:06:56.9495665Z OK
2022-05-18T04:06:56.9495756Z 
2022-05-18T04:06:56.9495839Z Generating XML reports...
2022-05-18T04:06:56.9530363Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040655.xml
2022-05-18T04:06:57.7221865Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmv2g8gt9
2022-05-18T04:06:57.7222337Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmv2g8gt9/_remote_module_non_scriptable.py
2022-05-18T04:06:57.9738070Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:06:57.9748468Z 
2022-05-18T04:06:57.9748693Z Running tests...
2022-05-18T04:06:58.2899177Z ----------------------------------------------------------------------
2022-05-18T04:06:58.2899957Z   test_rref_proxy_tensor (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8729
2022-05-18T04:06:58.2922891Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8730
2022-05-18T04:06:58.2946363Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8731
2022-05-18T04:06:58.2970974Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8732
2022-05-18T04:06:58.9809271Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmfpbxp0e
2022-05-18T04:06:58.9823217Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmfpbxp0e/_remote_module_non_scriptable.py
2022-05-18T04:06:58.9957381Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgmv58f2x
2022-05-18T04:06:58.9958903Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgmv58f2x/_remote_module_non_scriptable.py
2022-05-18T04:06:59.0006316Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkokqybjo
2022-05-18T04:06:59.0008135Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkokqybjo/_remote_module_non_scriptable.py
2022-05-18T04:06:59.0255483Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp89a7oe7x
2022-05-18T04:06:59.0256404Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp89a7oe7x/_remote_module_non_scriptable.py
2022-05-18T04:06:59.2294217Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:06:59.2457523Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:06:59.2493541Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:06:59.2735671Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:06:59.8013796Z ok (1.826s)
2022-05-18T04:06:59.8058755Z 
2022-05-18T04:06:59.8059636Z ----------------------------------------------------------------------
2022-05-18T04:06:59.8060291Z Ran 1 test in 1.826s
2022-05-18T04:06:59.8062595Z 
2022-05-18T04:06:59.8063754Z OK
2022-05-18T04:06:59.8063955Z 
2022-05-18T04:06:59.8064276Z Generating XML reports...
2022-05-18T04:06:59.8065536Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040657.xml
2022-05-18T04:07:00.5793305Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9wljej32
2022-05-18T04:07:00.5794015Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9wljej32/_remote_module_non_scriptable.py
2022-05-18T04:07:00.8350026Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:07:00.8359239Z 
2022-05-18T04:07:00.8359355Z Running tests...
2022-05-18T04:07:00.8359950Z ----------------------------------------------------------------------
2022-05-18T04:07:01.1516707Z   test_rref_proxy_tensor_self (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8948
2022-05-18T04:07:01.1538434Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8949
2022-05-18T04:07:01.1561952Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8950
2022-05-18T04:07:01.1586798Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8951
2022-05-18T04:07:01.8161813Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbr8irq7h
2022-05-18T04:07:01.8162571Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbr8irq7h/_remote_module_non_scriptable.py
2022-05-18T04:07:01.8262316Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk6dfwtmy
2022-05-18T04:07:01.8263644Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk6dfwtmy/_remote_module_non_scriptable.py
2022-05-18T04:07:01.8910946Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3nklsd2u
2022-05-18T04:07:01.8911694Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3nklsd2u/_remote_module_non_scriptable.py
2022-05-18T04:07:01.9036205Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzglt1bmw
2022-05-18T04:07:01.9037594Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzglt1bmw/_remote_module_non_scriptable.py
2022-05-18T04:07:02.0636803Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:07:02.0729495Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:07:02.1379296Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:07:02.1522294Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:07:02.6629701Z ok (1.827s)
2022-05-18T04:07:02.6629956Z 
2022-05-18T04:07:02.6630457Z ----------------------------------------------------------------------
2022-05-18T04:07:02.6630911Z Ran 1 test in 1.827s
2022-05-18T04:07:02.6631039Z 
2022-05-18T04:07:02.6631101Z OK
2022-05-18T04:07:02.6631192Z 
2022-05-18T04:07:02.6631283Z Generating XML reports...
2022-05-18T04:07:02.6664155Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040700.xml
2022-05-18T04:07:03.4325552Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3ss20fsh
2022-05-18T04:07:03.4326319Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3ss20fsh/_remote_module_non_scriptable.py
2022-05-18T04:07:03.6863658Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:07:03.6872410Z 
2022-05-18T04:07:03.6872501Z Running tests...
2022-05-18T04:07:03.6873004Z ----------------------------------------------------------------------
2022-05-18T04:07:04.0016451Z   test_rref_py_pickle_not_supported (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9167
2022-05-18T04:07:04.0039481Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9168
2022-05-18T04:07:04.0063396Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9169
2022-05-18T04:07:04.0088141Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9170
2022-05-18T04:07:04.6873326Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptfctrh27
2022-05-18T04:07:04.6874091Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptfctrh27/_remote_module_non_scriptable.py
2022-05-18T04:07:04.6899285Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp6gjenz5
2022-05-18T04:07:04.6900754Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp6gjenz5/_remote_module_non_scriptable.py
2022-05-18T04:07:04.6986879Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0zhgvb79
2022-05-18T04:07:04.6988465Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0zhgvb79/_remote_module_non_scriptable.py
2022-05-18T04:07:04.7156266Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkrwr08zg
2022-05-18T04:07:04.7157343Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkrwr08zg/_remote_module_non_scriptable.py
2022-05-18T04:07:04.9371466Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:07:04.9373637Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:07:04.9471631Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:07:04.9644351Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:07:05.4128454Z ok (1.725s)
2022-05-18T04:07:05.4128712Z 
2022-05-18T04:07:05.4129275Z ----------------------------------------------------------------------
2022-05-18T04:07:05.4129540Z Ran 1 test in 1.725s
2022-05-18T04:07:05.4129667Z 
2022-05-18T04:07:05.4129716Z OK
2022-05-18T04:07:05.4129808Z 
2022-05-18T04:07:05.4129900Z Generating XML reports...
2022-05-18T04:07:05.4163702Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040703.xml
2022-05-18T04:07:06.1865825Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv0zrktia
2022-05-18T04:07:06.1866574Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv0zrktia/_remote_module_non_scriptable.py
2022-05-18T04:07:06.4414836Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:07:06.4425186Z 
2022-05-18T04:07:06.4425613Z Running tests...
2022-05-18T04:07:06.4426015Z ----------------------------------------------------------------------
2022-05-18T04:07:06.7591857Z   test_rref_str (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9386
2022-05-18T04:07:06.7615530Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9387
2022-05-18T04:07:06.7638746Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9388
2022-05-18T04:07:06.7662810Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9389
2022-05-18T04:07:07.4395138Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7r7baam9
2022-05-18T04:07:07.4396438Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7r7baam9/_remote_module_non_scriptable.py
2022-05-18T04:07:07.4863981Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph8rbno_9
2022-05-18T04:07:07.4864730Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph8rbno_9/_remote_module_non_scriptable.py
2022-05-18T04:07:07.4938813Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_pz2xocx
2022-05-18T04:07:07.4940217Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_pz2xocx/_remote_module_non_scriptable.py
2022-05-18T04:07:07.5051031Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8eaqegnr
2022-05-18T04:07:07.5051993Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8eaqegnr/_remote_module_non_scriptable.py
2022-05-18T04:07:07.6877921Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:07:07.7343521Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:07:07.7386649Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:07:07.7518413Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:07:08.1701846Z ok (1.727s)
2022-05-18T04:07:08.1702142Z 
2022-05-18T04:07:08.1702668Z ----------------------------------------------------------------------
2022-05-18T04:07:08.1703201Z Ran 1 test in 1.728s
2022-05-18T04:07:08.1703322Z 
2022-05-18T04:07:08.1703383Z OK
2022-05-18T04:07:08.1703478Z 
2022-05-18T04:07:08.1703562Z Generating XML reports...
2022-05-18T04:07:08.1739216Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040706.xml
2022-05-18T04:07:08.9585569Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb36a4oau
2022-05-18T04:07:08.9586593Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb36a4oau/_remote_module_non_scriptable.py
2022-05-18T04:07:09.2127497Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:07:09.2136660Z 
2022-05-18T04:07:09.2136736Z Running tests...
2022-05-18T04:07:09.2137420Z ----------------------------------------------------------------------
2022-05-18T04:07:09.5290820Z   test_rref_timeout (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9605
2022-05-18T04:07:09.5313930Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9606
2022-05-18T04:07:09.5337421Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9607
2022-05-18T04:07:09.5363496Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9608
2022-05-18T04:07:10.1719014Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8vfrqze5
2022-05-18T04:07:10.1719772Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnqosmika
2022-05-18T04:07:10.1721118Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8vfrqze5/_remote_module_non_scriptable.py
2022-05-18T04:07:10.1721829Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnqosmika/_remote_module_non_scriptable.py
2022-05-18T04:07:10.2035183Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpabzjfxue
2022-05-18T04:07:10.2035969Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpabzjfxue/_remote_module_non_scriptable.py
2022-05-18T04:07:10.2098357Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_lppbcz3
2022-05-18T04:07:10.2099752Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_lppbcz3/_remote_module_non_scriptable.py
2022-05-18T04:07:10.4190466Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:07:10.4210225Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:07:10.4504973Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:07:10.4596390Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:07:12.9433950Z ok (3.729s)
2022-05-18T04:07:12.9434296Z 
2022-05-18T04:07:12.9434786Z ----------------------------------------------------------------------
2022-05-18T04:07:12.9435040Z Ran 1 test in 3.730s
2022-05-18T04:07:12.9435156Z 
2022-05-18T04:07:12.9435220Z OK
2022-05-18T04:07:12.9435299Z 
2022-05-18T04:07:12.9435393Z Generating XML reports...
2022-05-18T04:07:12.9469006Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040709.xml
2022-05-18T04:07:13.7189211Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5tqtw5j8
2022-05-18T04:07:13.7189764Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5tqtw5j8/_remote_module_non_scriptable.py
2022-05-18T04:07:13.9719691Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:07:13.9728900Z 
2022-05-18T04:07:13.9729024Z Running tests...
2022-05-18T04:07:13.9729523Z ----------------------------------------------------------------------
2022-05-18T04:07:14.2878743Z   test_rref_type_blocking (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9824
2022-05-18T04:07:14.2901478Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9825
2022-05-18T04:07:14.2924451Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9826
2022-05-18T04:07:14.2949434Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9827
2022-05-18T04:07:14.9108160Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpovzvo4qh
2022-05-18T04:07:14.9109140Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpovzvo4qh/_remote_module_non_scriptable.py
2022-05-18T04:07:14.9380150Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7_vwoflz
2022-05-18T04:07:14.9381123Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7_vwoflz/_remote_module_non_scriptable.py
2022-05-18T04:07:14.9596000Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjukttjz8
2022-05-18T04:07:14.9596719Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjukttjz8/_remote_module_non_scriptable.py
2022-05-18T04:07:14.9678979Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp54owdyd_
2022-05-18T04:07:14.9680528Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp54owdyd_/_remote_module_non_scriptable.py
2022-05-18T04:07:15.1608476Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:07:15.1845507Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:07:15.2100851Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:07:15.2158997Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:07:15.6988939Z ok (1.726s)
2022-05-18T04:07:15.6989218Z 
2022-05-18T04:07:15.6989732Z ----------------------------------------------------------------------
2022-05-18T04:07:15.6989975Z Ran 1 test in 1.726s
2022-05-18T04:07:15.6990096Z 
2022-05-18T04:07:15.6990158Z OK
2022-05-18T04:07:15.6990251Z 
2022-05-18T04:07:15.6990349Z Generating XML reports...
2022-05-18T04:07:15.7023817Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040713.xml
2022-05-18T04:07:16.4687099Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphvjd3jpq
2022-05-18T04:07:16.4687779Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphvjd3jpq/_remote_module_non_scriptable.py
2022-05-18T04:07:16.7232712Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:07:16.7242294Z 
2022-05-18T04:07:16.7242417Z Running tests...
2022-05-18T04:07:16.7242855Z ----------------------------------------------------------------------
2022-05-18T04:07:17.0426885Z   test_rref_type_non_blocking (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10043
2022-05-18T04:07:17.0449897Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10044
2022-05-18T04:07:17.0473882Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10045
2022-05-18T04:07:17.0498406Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10046
2022-05-18T04:07:17.7225323Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa2yln0_7
2022-05-18T04:07:17.7226146Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa2yln0_7/_remote_module_non_scriptable.py
2022-05-18T04:07:17.7229842Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsjcgpfwx
2022-05-18T04:07:17.7232377Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsjcgpfwx/_remote_module_non_scriptable.py
2022-05-18T04:07:17.7425640Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnwo_2w8m
2022-05-18T04:07:17.7428103Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnwo_2w8m/_remote_module_non_scriptable.py
2022-05-18T04:07:17.7818788Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpiqjc66f0
2022-05-18T04:07:17.7819981Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpiqjc66f0/_remote_module_non_scriptable.py
2022-05-18T04:07:17.9670159Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:07:17.9712791Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:07:17.9900917Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:07:18.0287450Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:07:18.4539491Z ok (1.729s)
2022-05-18T04:07:18.4539749Z 
2022-05-18T04:07:18.4540396Z ----------------------------------------------------------------------
2022-05-18T04:07:18.4540873Z Ran 1 test in 1.730s
2022-05-18T04:07:18.4541004Z 
2022-05-18T04:07:18.4541067Z OK
2022-05-18T04:07:18.4541146Z 
2022-05-18T04:07:18.4541246Z Generating XML reports...
2022-05-18T04:07:18.4576905Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040716.xml
2022-05-18T04:07:19.2271775Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3bpyxuew
2022-05-18T04:07:19.2272467Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3bpyxuew/_remote_module_non_scriptable.py
2022-05-18T04:07:19.4792733Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:07:19.4803250Z 
2022-05-18T04:07:19.4803559Z Running tests...
2022-05-18T04:07:19.4803945Z ----------------------------------------------------------------------
2022-05-18T04:07:19.7919748Z   test_rref_type_owner_blocking (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10262
2022-05-18T04:07:19.7942429Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10263
2022-05-18T04:07:19.7965756Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10264
2022-05-18T04:07:19.7990445Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10265
2022-05-18T04:07:20.4262298Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyzt65wn0
2022-05-18T04:07:20.4263817Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyzt65wn0/_remote_module_non_scriptable.py
2022-05-18T04:07:20.4482594Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzwc55tsl
2022-05-18T04:07:20.4483782Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzwc55tsl/_remote_module_non_scriptable.py
2022-05-18T04:07:20.4507312Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5cd7g8sf
2022-05-18T04:07:20.4508851Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5cd7g8sf/_remote_module_non_scriptable.py
2022-05-18T04:07:20.4549685Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0ybdsbqk
2022-05-18T04:07:20.4551418Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0ybdsbqk/_remote_module_non_scriptable.py
2022-05-18T04:07:20.6725905Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:07:20.6967373Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:07:20.6977015Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:07:20.7049588Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:07:21.2031435Z ok (1.723s)
2022-05-18T04:07:21.2031716Z 
2022-05-18T04:07:21.2032169Z ----------------------------------------------------------------------
2022-05-18T04:07:21.2032423Z Ran 1 test in 1.723s
2022-05-18T04:07:21.2032539Z 
2022-05-18T04:07:21.2032603Z OK
2022-05-18T04:07:21.2032696Z 
2022-05-18T04:07:21.2032798Z Generating XML reports...
2022-05-18T04:07:21.2066124Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040719.xml
2022-05-18T04:07:21.9665745Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa1zw940j
2022-05-18T04:07:21.9667092Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa1zw940j/_remote_module_non_scriptable.py
2022-05-18T04:07:22.2209304Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:07:22.2219524Z 
2022-05-18T04:07:22.2219946Z Running tests...
2022-05-18T04:07:22.2220346Z ----------------------------------------------------------------------
2022-05-18T04:07:22.5457655Z   test_rref_type_owner_non_blocking (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10481
2022-05-18T04:07:22.5480936Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10482
2022-05-18T04:07:22.5504146Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10483
2022-05-18T04:07:22.5528696Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10484
2022-05-18T04:07:23.1814575Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8t70pj1i
2022-05-18T04:07:23.1815713Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8t70pj1i/_remote_module_non_scriptable.py
2022-05-18T04:07:23.1914130Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwjwl_nyb
2022-05-18T04:07:23.1915110Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwjwl_nyb/_remote_module_non_scriptable.py
2022-05-18T04:07:23.2003388Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph_fvpz78
2022-05-18T04:07:23.2004488Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo2zp8i8n
2022-05-18T04:07:23.2005574Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph_fvpz78/_remote_module_non_scriptable.py
2022-05-18T04:07:23.2006314Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo2zp8i8n/_remote_module_non_scriptable.py
2022-05-18T04:07:23.4372892Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:07:23.4464519Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:07:23.4564022Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:07:23.4566041Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:07:23.9569179Z ok (1.735s)
2022-05-18T04:07:23.9570542Z 
2022-05-18T04:07:23.9570889Z ----------------------------------------------------------------------
2022-05-18T04:07:23.9571145Z Ran 1 test in 1.735s
2022-05-18T04:07:23.9571260Z 
2022-05-18T04:07:23.9571307Z OK
2022-05-18T04:07:23.9571401Z 
2022-05-18T04:07:23.9571494Z Generating XML reports...
2022-05-18T04:07:23.9604868Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040722.xml
2022-05-18T04:07:24.7366474Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqr4_hkxv
2022-05-18T04:07:24.7367312Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqr4_hkxv/_remote_module_non_scriptable.py
2022-05-18T04:07:24.9930968Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:07:24.9941423Z 
2022-05-18T04:07:24.9941892Z Running tests...
2022-05-18T04:07:24.9942502Z ----------------------------------------------------------------------
2022-05-18T04:07:25.3203575Z   test_rref_type_slow_init (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10700
2022-05-18T04:07:25.3226578Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10701
2022-05-18T04:07:25.3250312Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10702
2022-05-18T04:07:25.3275520Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10703
2022-05-18T04:07:25.9868723Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppb8s6chr
2022-05-18T04:07:25.9871476Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppb8s6chr/_remote_module_non_scriptable.py
2022-05-18T04:07:25.9946556Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0h9wp7kn
2022-05-18T04:07:25.9947509Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0h9wp7kn/_remote_module_non_scriptable.py
2022-05-18T04:07:26.0450883Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjckkcz71
2022-05-18T04:07:26.0451790Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjckkcz71/_remote_module_non_scriptable.py
2022-05-18T04:07:26.0541046Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9zvsq3_m
2022-05-18T04:07:26.0542118Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9zvsq3_m/_remote_module_non_scriptable.py
2022-05-18T04:07:26.2426869Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:07:26.2501948Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:07:26.2986114Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:07:26.3093923Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:07:27.8334781Z ok (2.839s)
2022-05-18T04:07:27.8335055Z 
2022-05-18T04:07:27.8335569Z ----------------------------------------------------------------------
2022-05-18T04:07:27.8335838Z Ran 1 test in 2.839s
2022-05-18T04:07:27.8335956Z 
2022-05-18T04:07:27.8336005Z OK
2022-05-18T04:07:27.8336097Z 
2022-05-18T04:07:27.8336191Z Generating XML reports...
2022-05-18T04:07:27.8369223Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040724.xml
2022-05-18T04:07:28.6257339Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2cy_cmot
2022-05-18T04:07:28.6257802Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2cy_cmot/_remote_module_non_scriptable.py
2022-05-18T04:07:28.8776795Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:07:28.8786712Z 
2022-05-18T04:07:28.8786855Z Running tests...
2022-05-18T04:07:28.8787269Z ----------------------------------------------------------------------
2022-05-18T04:07:29.1982311Z   test_rref_type_with_error_blocking (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10919
2022-05-18T04:07:29.2005268Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10920
2022-05-18T04:07:29.2029894Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10921
2022-05-18T04:07:29.2055187Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10922
2022-05-18T04:07:29.8377885Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg85puz2r
2022-05-18T04:07:29.8378663Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg85puz2r/_remote_module_non_scriptable.py
2022-05-18T04:07:29.8412861Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1yev337q
2022-05-18T04:07:29.8414823Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1yev337q/_remote_module_non_scriptable.py
2022-05-18T04:07:29.8625663Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr98gli0q
2022-05-18T04:07:29.8626472Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr98gli0q/_remote_module_non_scriptable.py
2022-05-18T04:07:29.8716587Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppptnprku
2022-05-18T04:07:29.8717945Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppptnprku/_remote_module_non_scriptable.py
2022-05-18T04:07:30.0898046Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:07:30.0919372Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:07:30.1141737Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:07:30.1216169Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:07:30.3255846Z On WorkerInfo(id=1, name=worker1):
2022-05-18T04:07:30.3256628Z ValueError('Expected error')
2022-05-18T04:07:30.3257074Z Traceback (most recent call last):
2022-05-18T04:07:30.3258005Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function
2022-05-18T04:07:30.3258758Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-05-18T04:07:30.3259724Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 396, in raise_func
2022-05-18T04:07:30.3260435Z     raise ValueError(expected_err)
2022-05-18T04:07:30.3260901Z ValueError: Expected error
2022-05-18T04:07:30.3261173Z 
2022-05-18T04:07:30.3269767Z On WorkerInfo(id=1, name=worker1):
2022-05-18T04:07:30.3272097Z ValueError('On WorkerInfo(id=1, name=worker1):\nValueError(\'Expected error\')\nTraceback (most recent call last):\n  File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function\n    result = python_udf.func(*python_udf.args, **python_udf.kwargs)\n  File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 396, in raise_func\n    raise ValueError(expected_err)\nValueError: Expected error\n')
2022-05-18T04:07:30.3273603Z Traceback (most recent call last):
2022-05-18T04:07:30.3274467Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function
2022-05-18T04:07:30.3275220Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-05-18T04:07:30.3276145Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/api.py", line 429, in _rref_typeof_on_owner
2022-05-18T04:07:30.3276790Z     rref_type = type(rref.local_value())
2022-05-18T04:07:30.3277672Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 220, in _handle_exception
2022-05-18T04:07:30.3278634Z     raise result.exception_type(result.msg.encode("utf-8").decode("unicode_escape"))
2022-05-18T04:07:30.3279252Z ValueError: On WorkerInfo(id=1, name=worker1):
2022-05-18T04:07:30.3279810Z ValueError('Expected error')
2022-05-18T04:07:30.3280260Z Traceback (most recent call last):
2022-05-18T04:07:30.3281084Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function
2022-05-18T04:07:30.3281796Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-05-18T04:07:30.3282711Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 396, in raise_func
2022-05-18T04:07:30.3283220Z     raise ValueError(expected_err)
2022-05-18T04:07:30.3283655Z ValueError: Expected error
2022-05-18T04:07:30.3283922Z 
2022-05-18T04:07:30.3283932Z 
2022-05-18T04:07:30.3410582Z On WorkerInfo(id=2, name=worker2):
2022-05-18T04:07:30.3414142Z ValueError('Expected error')
2022-05-18T04:07:30.3414666Z Traceback (most recent call last):
2022-05-18T04:07:30.3415549Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function
2022-05-18T04:07:30.3416253Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-05-18T04:07:30.3417243Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 396, in raise_func
2022-05-18T04:07:30.3417894Z     raise ValueError(expected_err)
2022-05-18T04:07:30.3418320Z ValueError: Expected error
2022-05-18T04:07:30.3418973Z 
2022-05-18T04:07:30.3422282Z On WorkerInfo(id=2, name=worker2):
2022-05-18T04:07:30.3424561Z ValueError('On WorkerInfo(id=2, name=worker2):\nValueError(\'Expected error\')\nTraceback (most recent call last):\n  File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function\n    result = python_udf.func(*python_udf.args, **python_udf.kwargs)\n  File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 396, in raise_func\n    raise ValueError(expected_err)\nValueError: Expected error\n')
2022-05-18T04:07:30.3425686Z Traceback (most recent call last):
2022-05-18T04:07:30.3426389Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function
2022-05-18T04:07:30.3426955Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-05-18T04:07:30.3427667Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/api.py", line 429, in _rref_typeof_on_owner
2022-05-18T04:07:30.3428192Z     rref_type = type(rref.local_value())
2022-05-18T04:07:30.3428873Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 220, in _handle_exception
2022-05-18T04:07:30.3429604Z     raise result.exception_type(result.msg.encode("utf-8").decode("unicode_escape"))
2022-05-18T04:07:30.3430079Z ValueError: On WorkerInfo(id=2, name=worker2):
2022-05-18T04:07:30.3430521Z ValueError('Expected error')
2022-05-18T04:07:30.3430898Z Traceback (most recent call last):
2022-05-18T04:07:30.3431534Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function
2022-05-18T04:07:30.3432093Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-05-18T04:07:30.3432851Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 396, in raise_func
2022-05-18T04:07:30.3433359Z     raise ValueError(expected_err)
2022-05-18T04:07:30.3433713Z ValueError: Expected error
2022-05-18T04:07:30.3433895Z 
2022-05-18T04:07:30.3433902Z 
2022-05-18T04:07:30.3452691Z On WorkerInfo(id=0, name=worker0):
2022-05-18T04:07:30.3453212Z ValueError('Expected error')
2022-05-18T04:07:30.3453581Z Traceback (most recent call last):
2022-05-18T04:07:30.3454266Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function
2022-05-18T04:07:30.3454837Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-05-18T04:07:30.3455598Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 396, in raise_func
2022-05-18T04:07:30.3456097Z     raise ValueError(expected_err)
2022-05-18T04:07:30.3456461Z ValueError: Expected error
2022-05-18T04:07:30.3456674Z 
2022-05-18T04:07:30.3462087Z On WorkerInfo(id=0, name=worker0):
2022-05-18T04:07:30.3464225Z ValueError('On WorkerInfo(id=0, name=worker0):\nValueError(\'Expected error\')\nTraceback (most recent call last):\n  File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function\n    result = python_udf.func(*python_udf.args, **python_udf.kwargs)\n  File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 396, in raise_func\n    raise ValueError(expected_err)\nValueError: Expected error\n')
2022-05-18T04:07:30.3465398Z Traceback (most recent call last):
2022-05-18T04:07:30.3466057Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function
2022-05-18T04:07:30.3466620Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-05-18T04:07:30.3467369Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/api.py", line 429, in _rref_typeof_on_owner
2022-05-18T04:07:30.3467856Z     rref_type = type(rref.local_value())
2022-05-18T04:07:30.3468536Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 220, in _handle_exception
2022-05-18T04:07:30.3469484Z     raise result.exception_type(result.msg.encode("utf-8").decode("unicode_escape"))
2022-05-18T04:07:30.3469956Z ValueError: On WorkerInfo(id=0, name=worker0):
2022-05-18T04:07:30.3470481Z ValueError('Expected error')
2022-05-18T04:07:30.3470853Z Traceback (most recent call last):
2022-05-18T04:07:30.3471510Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function
2022-05-18T04:07:30.3472056Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-05-18T04:07:30.3472814Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 396, in raise_func
2022-05-18T04:07:30.3473328Z     raise ValueError(expected_err)
2022-05-18T04:07:30.3473678Z ValueError: Expected error
2022-05-18T04:07:30.3473901Z 
2022-05-18T04:07:30.3473909Z 
2022-05-18T04:07:30.3756846Z On WorkerInfo(id=3, name=worker3):
2022-05-18T04:07:30.3757633Z ValueError('Expected error')
2022-05-18T04:07:30.3758190Z Traceback (most recent call last):
2022-05-18T04:07:30.3759027Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function
2022-05-18T04:07:30.3759778Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-05-18T04:07:30.3760705Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 396, in raise_func
2022-05-18T04:07:30.3761365Z     raise ValueError(expected_err)
2022-05-18T04:07:30.3761742Z ValueError: Expected error
2022-05-18T04:07:30.3762067Z 
2022-05-18T04:07:30.3762255Z On WorkerInfo(id=3, name=worker3):
2022-05-18T04:07:30.3763953Z ValueError('On WorkerInfo(id=3, name=worker3):\nValueError(\'Expected error\')\nTraceback (most recent call last):\n  File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function\n    result = python_udf.func(*python_udf.args, **python_udf.kwargs)\n  File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 396, in raise_func\n    raise ValueError(expected_err)\nValueError: Expected error\n')
2022-05-18T04:07:30.3765049Z Traceback (most recent call last):
2022-05-18T04:07:30.3765697Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function
2022-05-18T04:07:30.3766267Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-05-18T04:07:30.3766956Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/api.py", line 429, in _rref_typeof_on_owner
2022-05-18T04:07:30.3767420Z     rref_type = type(rref.local_value())
2022-05-18T04:07:30.3768106Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 220, in _handle_exception
2022-05-18T04:07:30.3768930Z     raise result.exception_type(result.msg.encode("utf-8").decode("unicode_escape"))
2022-05-18T04:07:30.3769438Z ValueError: On WorkerInfo(id=3, name=worker3):
2022-05-18T04:07:30.3769889Z ValueError('Expected error')
2022-05-18T04:07:30.3770259Z Traceback (most recent call last):
2022-05-18T04:07:30.3770932Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function
2022-05-18T04:07:30.3771482Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-05-18T04:07:30.3772245Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 396, in raise_func
2022-05-18T04:07:30.3772746Z     raise ValueError(expected_err)
2022-05-18T04:07:30.3773126Z ValueError: Expected error
2022-05-18T04:07:30.3773320Z 
2022-05-18T04:07:30.3773327Z 
2022-05-18T04:07:30.6095677Z ok (1.731s)
2022-05-18T04:07:30.6095837Z 
2022-05-18T04:07:30.6096254Z ----------------------------------------------------------------------
2022-05-18T04:07:30.6096693Z Ran 1 test in 1.731s
2022-05-18T04:07:30.6096904Z 
2022-05-18T04:07:30.6097003Z OK
2022-05-18T04:07:30.6097114Z 
2022-05-18T04:07:30.6097415Z Generating XML reports...
2022-05-18T04:07:30.6132424Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040728.xml
2022-05-18T04:07:31.4038290Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptkse9cyx
2022-05-18T04:07:31.4038790Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptkse9cyx/_remote_module_non_scriptable.py
2022-05-18T04:07:31.6611955Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:07:31.6621071Z 
2022-05-18T04:07:31.6621280Z Running tests...
2022-05-18T04:07:31.6621625Z ----------------------------------------------------------------------
2022-05-18T04:07:31.9855119Z   test_rref_type_with_error_non_blocking (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11138
2022-05-18T04:07:31.9878170Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11139
2022-05-18T04:07:31.9901058Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11140
2022-05-18T04:07:31.9925685Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11141
2022-05-18T04:07:32.6853547Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphiew8mc8
2022-05-18T04:07:32.6854731Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphiew8mc8/_remote_module_non_scriptable.py
2022-05-18T04:07:32.6905868Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpopwgnz7b
2022-05-18T04:07:32.6906808Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpopwgnz7b/_remote_module_non_scriptable.py
2022-05-18T04:07:32.6965468Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5kgktvtb
2022-05-18T04:07:32.6967023Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5kgktvtb/_remote_module_non_scriptable.py
2022-05-18T04:07:32.7249164Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2l0b35h8
2022-05-18T04:07:32.7250378Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2l0b35h8/_remote_module_non_scriptable.py
2022-05-18T04:07:32.9359680Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:07:32.9399622Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:07:32.9476315Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:07:32.9743769Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:07:33.1650183Z On WorkerInfo(id=1, name=worker1):
2022-05-18T04:07:33.1650953Z ValueError('Expected error')
2022-05-18T04:07:33.1651873Z Traceback (most recent call last):
2022-05-18T04:07:33.1652909Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function
2022-05-18T04:07:33.1653651Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-05-18T04:07:33.1654659Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 396, in raise_func
2022-05-18T04:07:33.1655319Z     raise ValueError(expected_err)
2022-05-18T04:07:33.1655751Z ValueError: Expected error
2022-05-18T04:07:33.1656018Z 
2022-05-18T04:07:33.1659523Z On WorkerInfo(id=1, name=worker1):
2022-05-18T04:07:33.1668448Z ValueError('On WorkerInfo(id=1, name=worker1):\nValueError(\'Expected error\')\nTraceback (most recent call last):\n  File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function\n    result = python_udf.func(*python_udf.args, **python_udf.kwargs)\n  File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 396, in raise_func\n    raise ValueError(expected_err)\nValueError: Expected error\n')
2022-05-18T04:07:33.1669165Z Traceback (most recent call last):
2022-05-18T04:07:33.1669804Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function
2022-05-18T04:07:33.1670201Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-05-18T04:07:33.1670618Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/api.py", line 429, in _rref_typeof_on_owner
2022-05-18T04:07:33.1670916Z     rref_type = type(rref.local_value())
2022-05-18T04:07:33.1671311Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 220, in _handle_exception
2022-05-18T04:07:33.1671735Z     raise result.exception_type(result.msg.encode("utf-8").decode("unicode_escape"))
2022-05-18T04:07:33.1672009Z ValueError: On WorkerInfo(id=1, name=worker1):
2022-05-18T04:07:33.1672261Z ValueError('Expected error')
2022-05-18T04:07:33.1672473Z Traceback (most recent call last):
2022-05-18T04:07:33.1672841Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function
2022-05-18T04:07:33.1673177Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-05-18T04:07:33.1673618Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 396, in raise_func
2022-05-18T04:07:33.1673919Z     raise ValueError(expected_err)
2022-05-18T04:07:33.1674116Z ValueError: Expected error
2022-05-18T04:07:33.1674239Z 
2022-05-18T04:07:33.1674243Z 
2022-05-18T04:07:33.1832380Z On WorkerInfo(id=2, name=worker2):
2022-05-18T04:07:33.1833238Z ValueError('Expected error')
2022-05-18T04:07:33.1833946Z Traceback (most recent call last):
2022-05-18T04:07:33.1834738Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function
2022-05-18T04:07:33.1835311Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-05-18T04:07:33.1836067Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 396, in raise_func
2022-05-18T04:07:33.1836716Z     raise ValueError(expected_err)
2022-05-18T04:07:33.1837091Z ValueError: Expected error
2022-05-18T04:07:33.1837311Z 
2022-05-18T04:07:33.1843165Z On WorkerInfo(id=2, name=worker2):
2022-05-18T04:07:33.1846570Z ValueError('On WorkerInfo(id=2, name=worker2):\nValueError(\'Expected error\')\nTraceback (most recent call last):\n  File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function\n    result = python_udf.func(*python_udf.args, **python_udf.kwargs)\n  File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 396, in raise_func\n    raise ValueError(expected_err)\nValueError: Expected error\n')
2022-05-18T04:07:33.1849814Z Traceback (most recent call last):
2022-05-18T04:07:33.1852422Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function
2022-05-18T04:07:33.1853207Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-05-18T04:07:33.1854055Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/api.py", line 429, in _rref_typeof_on_owner
2022-05-18T04:07:33.1856156Z     rref_type = type(rref.local_value())
2022-05-18T04:07:33.1858349Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 220, in _handle_exception
2022-05-18T04:07:33.1859374Z     raise result.exception_type(result.msg.encode("utf-8").decode("unicode_escape"))
2022-05-18T04:07:33.1860020Z ValueError: On WorkerInfo(id=2, name=worker2):
2022-05-18T04:07:33.1860477Z ValueError('Expected error')
2022-05-18T04:07:33.1860860Z Traceback (most recent call last):
2022-05-18T04:07:33.1861694Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function
2022-05-18T04:07:33.1862461Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-05-18T04:07:33.1863639Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 396, in raise_func
2022-05-18T04:07:33.1866090Z     raise ValueError(expected_err)
2022-05-18T04:07:33.1866554Z ValueError: Expected error
2022-05-18T04:07:33.1866792Z 
2022-05-18T04:07:33.1866935Z 
2022-05-18T04:07:33.1867105Z On WorkerInfo(id=0, name=worker0):
2022-05-18T04:07:33.1867554Z ValueError('Expected error')
2022-05-18T04:07:33.1867898Z Traceback (most recent call last):
2022-05-18T04:07:33.1868615Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function
2022-05-18T04:07:33.1869198Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-05-18T04:07:33.1869997Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 396, in raise_func
2022-05-18T04:07:33.1870563Z     raise ValueError(expected_err)
2022-05-18T04:07:33.1870856Z ValueError: Expected error
2022-05-18T04:07:33.1870978Z 
2022-05-18T04:07:33.1871065Z On WorkerInfo(id=0, name=worker0):
2022-05-18T04:07:33.1872060Z ValueError('On WorkerInfo(id=0, name=worker0):\nValueError(\'Expected error\')\nTraceback (most recent call last):\n  File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function\n    result = python_udf.func(*python_udf.args, **python_udf.kwargs)\n  File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 396, in raise_func\n    raise ValueError(expected_err)\nValueError: Expected error\n')
2022-05-18T04:07:33.1872724Z Traceback (most recent call last):
2022-05-18T04:07:33.1873102Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function
2022-05-18T04:07:33.1873418Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-05-18T04:07:33.1873834Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/api.py", line 429, in _rref_typeof_on_owner
2022-05-18T04:07:33.1874125Z     rref_type = type(rref.local_value())
2022-05-18T04:07:33.1874504Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 220, in _handle_exception
2022-05-18T04:07:33.1874928Z     raise result.exception_type(result.msg.encode("utf-8").decode("unicode_escape"))
2022-05-18T04:07:33.1875224Z ValueError: On WorkerInfo(id=0, name=worker0):
2022-05-18T04:07:33.1875478Z ValueError('Expected error')
2022-05-18T04:07:33.1875675Z Traceback (most recent call last):
2022-05-18T04:07:33.1876051Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function
2022-05-18T04:07:33.1876376Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-05-18T04:07:33.1876801Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 396, in raise_func
2022-05-18T04:07:33.1877100Z     raise ValueError(expected_err)
2022-05-18T04:07:33.1877304Z ValueError: Expected error
2022-05-18T04:07:33.1877424Z 
2022-05-18T04:07:33.1877431Z 
2022-05-18T04:07:33.2312211Z On WorkerInfo(id=3, name=worker3):
2022-05-18T04:07:33.2312699Z ValueError('Expected error')
2022-05-18T04:07:33.2313076Z Traceback (most recent call last):
2022-05-18T04:07:33.2313796Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function
2022-05-18T04:07:33.2314362Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-05-18T04:07:33.2315082Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 396, in raise_func
2022-05-18T04:07:33.2315523Z     raise ValueError(expected_err)
2022-05-18T04:07:33.2315821Z ValueError: Expected error
2022-05-18T04:07:33.2316010Z 
2022-05-18T04:07:33.2316645Z On WorkerInfo(id=3, name=worker3):
2022-05-18T04:07:33.2327845Z ValueError('On WorkerInfo(id=3, name=worker3):\nValueError(\'Expected error\')\nTraceback (most recent call last):\n  File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function\n    result = python_udf.func(*python_udf.args, **python_udf.kwargs)\n  File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 396, in raise_func\n    raise ValueError(expected_err)\nValueError: Expected error\n')
2022-05-18T04:07:33.2328652Z Traceback (most recent call last):
2022-05-18T04:07:33.2329370Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function
2022-05-18T04:07:33.2329889Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-05-18T04:07:33.2330520Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/api.py", line 429, in _rref_typeof_on_owner
2022-05-18T04:07:33.2332379Z     rref_type = type(rref.local_value())
2022-05-18T04:07:33.2333368Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 220, in _handle_exception
2022-05-18T04:07:33.2334327Z     raise result.exception_type(result.msg.encode("utf-8").decode("unicode_escape"))
2022-05-18T04:07:33.2334975Z ValueError: On WorkerInfo(id=3, name=worker3):
2022-05-18T04:07:33.2335511Z ValueError('Expected error')
2022-05-18T04:07:33.2335974Z Traceback (most recent call last):
2022-05-18T04:07:33.2336850Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function
2022-05-18T04:07:33.2337563Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-05-18T04:07:33.2338550Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 396, in raise_func
2022-05-18T04:07:33.2339216Z     raise ValueError(expected_err)
2022-05-18T04:07:33.2339647Z ValueError: Expected error
2022-05-18T04:07:33.2339922Z 
2022-05-18T04:07:33.2339931Z 
2022-05-18T04:07:33.4967778Z ok (1.834s)
2022-05-18T04:07:33.4968006Z 
2022-05-18T04:07:33.4968464Z ----------------------------------------------------------------------
2022-05-18T04:07:33.4968921Z Ran 1 test in 1.835s
2022-05-18T04:07:33.4969122Z 
2022-05-18T04:07:33.4969197Z OK
2022-05-18T04:07:33.4969340Z 
2022-05-18T04:07:33.4969484Z Generating XML reports...
2022-05-18T04:07:33.5004032Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040731.xml
2022-05-18T04:07:34.2679555Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjwftvbe2
2022-05-18T04:07:34.2680146Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjwftvbe2/_remote_module_non_scriptable.py
2022-05-18T04:07:34.5244605Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:07:34.5254772Z 
2022-05-18T04:07:34.5254900Z Running tests...
2022-05-18T04:07:34.5255397Z ----------------------------------------------------------------------
2022-05-18T04:07:34.8662355Z   test_scalar_add (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11357
2022-05-18T04:07:34.8688944Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11358
2022-05-18T04:07:34.8713642Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11359
2022-05-18T04:07:34.8740294Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11360
2022-05-18T04:07:35.5323087Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp41ash_qk
2022-05-18T04:07:35.5323835Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp41ash_qk/_remote_module_non_scriptable.py
2022-05-18T04:07:35.5774604Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplo7e8lkg
2022-05-18T04:07:35.5775408Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplo7e8lkg/_remote_module_non_scriptable.py
2022-05-18T04:07:35.6034964Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzxttabv2
2022-05-18T04:07:35.6035704Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzxttabv2/_remote_module_non_scriptable.py
2022-05-18T04:07:35.6111726Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzjq3wno9
2022-05-18T04:07:35.6115194Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzjq3wno9/_remote_module_non_scriptable.py
2022-05-18T04:07:35.7975313Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:07:35.8406039Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:07:35.8664360Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:07:35.8720968Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:07:36.2780879Z ok (1.752s)
2022-05-18T04:07:36.2781152Z 
2022-05-18T04:07:36.2781663Z ----------------------------------------------------------------------
2022-05-18T04:07:36.2782010Z Ran 1 test in 1.752s
2022-05-18T04:07:36.2782133Z 
2022-05-18T04:07:36.2782193Z OK
2022-05-18T04:07:36.2782284Z 
2022-05-18T04:07:36.2782374Z Generating XML reports...
2022-05-18T04:07:36.2814755Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040734.xml
2022-05-18T04:07:37.0410905Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb5v51y__
2022-05-18T04:07:37.0411717Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb5v51y__/_remote_module_non_scriptable.py
2022-05-18T04:07:37.2943098Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:07:37.2952575Z 
2022-05-18T04:07:37.2952832Z Running tests...
2022-05-18T04:07:37.2953264Z ----------------------------------------------------------------------
2022-05-18T04:07:37.6083944Z   test_self_add (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11576
2022-05-18T04:07:37.6106610Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11577
2022-05-18T04:07:37.6129938Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11578
2022-05-18T04:07:37.6154594Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11579
2022-05-18T04:07:38.2700149Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprp4ui6gs
2022-05-18T04:07:38.2701194Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprp4ui6gs/_remote_module_non_scriptable.py
2022-05-18T04:07:38.2767688Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd_dcz9s7
2022-05-18T04:07:38.2769281Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd_dcz9s7/_remote_module_non_scriptable.py
2022-05-18T04:07:38.3088620Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa68bq0kd
2022-05-18T04:07:38.3089493Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa68bq0kd/_remote_module_non_scriptable.py
2022-05-18T04:07:38.3154103Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpco_dp46u
2022-05-18T04:07:38.3155331Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpco_dp46u/_remote_module_non_scriptable.py
2022-05-18T04:07:38.5183587Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:07:38.5224042Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:07:38.5564320Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:07:38.5624070Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:07:39.0195334Z ok (1.724s)
2022-05-18T04:07:39.0195524Z 
2022-05-18T04:07:39.0195837Z ----------------------------------------------------------------------
2022-05-18T04:07:39.0196154Z Ran 1 test in 1.724s
2022-05-18T04:07:39.0196513Z 
2022-05-18T04:07:39.0196575Z OK
2022-05-18T04:07:39.0196667Z 
2022-05-18T04:07:39.0196795Z Generating XML reports...
2022-05-18T04:07:39.0230623Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040737.xml
2022-05-18T04:07:39.7947220Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpigw4gsm9
2022-05-18T04:07:39.7948111Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpigw4gsm9/_remote_module_non_scriptable.py
2022-05-18T04:07:40.0521763Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:07:40.0531440Z 
2022-05-18T04:07:40.0531663Z Running tests...
2022-05-18T04:07:40.0532005Z ----------------------------------------------------------------------
2022-05-18T04:07:40.3806692Z   test_self_py_udf_remote (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11795
2022-05-18T04:07:40.3830118Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11796
2022-05-18T04:07:40.3853903Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11797
2022-05-18T04:07:40.3878475Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11798
2022-05-18T04:07:41.0250257Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwv6ea_se
2022-05-18T04:07:41.0250992Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwv6ea_se/_remote_module_non_scriptable.py
2022-05-18T04:07:41.0268982Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkguh61ib
2022-05-18T04:07:41.0269954Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkguh61ib/_remote_module_non_scriptable.py
2022-05-18T04:07:41.0499515Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd3ne1u3j
2022-05-18T04:07:41.0500264Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd3ne1u3j/_remote_module_non_scriptable.py
2022-05-18T04:07:41.0614297Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp19s7h01j
2022-05-18T04:07:41.0615296Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp19s7h01j/_remote_module_non_scriptable.py
2022-05-18T04:07:41.2758657Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:07:41.2765161Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:07:41.3032048Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:07:41.3117944Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:07:41.7921111Z ok (1.739s)
2022-05-18T04:07:41.7921321Z 
2022-05-18T04:07:41.7921680Z ----------------------------------------------------------------------
2022-05-18T04:07:41.7921921Z Ran 1 test in 1.739s
2022-05-18T04:07:41.7922038Z 
2022-05-18T04:07:41.7922118Z OK
2022-05-18T04:07:41.7922209Z 
2022-05-18T04:07:41.7922306Z Generating XML reports...
2022-05-18T04:07:41.7957357Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040740.xml
2022-05-18T04:07:42.6186012Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3151v6tm
2022-05-18T04:07:42.6187203Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3151v6tm/_remote_module_non_scriptable.py
2022-05-18T04:07:42.8826557Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:07:42.8836678Z 
2022-05-18T04:07:42.8836797Z Running tests...
2022-05-18T04:07:42.8837205Z ----------------------------------------------------------------------
2022-05-18T04:07:43.2255058Z   test_self_remote_rref_as_remote_arg (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12014
2022-05-18T04:07:43.2281304Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12015
2022-05-18T04:07:43.2305893Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12016
2022-05-18T04:07:43.2332985Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 12017
2022-05-18T04:07:43.8312541Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcq0jkgp8
2022-05-18T04:07:43.8313273Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcq0jkgp8/_remote_module_non_scriptable.py
2022-05-18T04:07:43.8328598Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp241e2wt2
2022-05-18T04:07:43.8331253Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp241e2wt2/_remote_module_non_scriptable.py
2022-05-18T04:07:43.8421830Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo9teac3x
2022-05-18T04:07:43.8423063Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo9teac3x/_remote_module_non_scriptable.py
2022-05-18T04:07:43.8910215Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo6z9xd1o
2022-05-18T04:07:43.8911095Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo6z9xd1o/_remote_module_non_scriptable.py
2022-05-18T04:07:44.0946123Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:07:44.0949192Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:07:44.1060465Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:07:44.1538289Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:07:44.7375781Z ok (1.854s)
2022-05-18T04:07:44.7376088Z 
2022-05-18T04:07:44.7376612Z ----------------------------------------------------------------------
2022-05-18T04:07:44.7377092Z Ran 1 test in 1.854s
2022-05-18T04:07:44.7377267Z 
2022-05-18T04:07:44.7377327Z OK
2022-05-18T04:07:44.7377407Z 
2022-05-18T04:07:44.7377503Z Generating XML reports...
2022-05-18T04:07:44.7411679Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040742.xml
2022-05-18T04:07:45.6178808Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph5tbujno
2022-05-18T04:07:45.6179984Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph5tbujno/_remote_module_non_scriptable.py
2022-05-18T04:07:45.8882437Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:07:45.8892853Z 
2022-05-18T04:07:45.8893032Z Running tests...
2022-05-18T04:07:45.8893444Z ----------------------------------------------------------------------
2022-05-18T04:07:46.2410168Z   test_self_remote_rref_as_rpc_arg (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12233
2022-05-18T04:07:46.2435736Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12234
2022-05-18T04:07:46.2461434Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12235
2022-05-18T04:07:46.2489407Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 12236
2022-05-18T04:07:46.9258824Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6z96harh
2022-05-18T04:07:46.9259558Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6z96harh/_remote_module_non_scriptable.py
2022-05-18T04:07:46.9260196Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0z5dfnxf
2022-05-18T04:07:46.9263403Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0z5dfnxf/_remote_module_non_scriptable.py
2022-05-18T04:07:46.9560068Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv0l3wfov
2022-05-18T04:07:46.9561384Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv0l3wfov/_remote_module_non_scriptable.py
2022-05-18T04:07:46.9820201Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk6ewy5_7
2022-05-18T04:07:46.9821214Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk6ewy5_7/_remote_module_non_scriptable.py
2022-05-18T04:07:47.1873631Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:07:47.1903657Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:07:47.2206551Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:07:47.2435697Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:07:47.8535016Z ok (1.964s)
2022-05-18T04:07:47.8535345Z 
2022-05-18T04:07:47.8535870Z ----------------------------------------------------------------------
2022-05-18T04:07:47.8536148Z Ran 1 test in 1.964s
2022-05-18T04:07:47.8536281Z 
2022-05-18T04:07:47.8536343Z OK
2022-05-18T04:07:47.8536423Z 
2022-05-18T04:07:47.8536526Z Generating XML reports...
2022-05-18T04:07:47.8570631Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040745.xml
2022-05-18T04:07:48.7392716Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp531i7u2y
2022-05-18T04:07:48.7393462Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp531i7u2y/_remote_module_non_scriptable.py
2022-05-18T04:07:49.0096595Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:07:49.0106640Z 
2022-05-18T04:07:49.0106773Z Running tests...
2022-05-18T04:07:49.0107361Z ----------------------------------------------------------------------
2022-05-18T04:07:49.3617432Z   test_self_remote_rref_as_self_remote_arg (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12452
2022-05-18T04:07:49.3643149Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12453
2022-05-18T04:07:49.3668581Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12454
2022-05-18T04:07:49.3693285Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 12455
2022-05-18T04:07:50.0705859Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpaa_3dmqo
2022-05-18T04:07:50.0707082Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpaa_3dmqo/_remote_module_non_scriptable.py
2022-05-18T04:07:50.1045031Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe5f6ertm
2022-05-18T04:07:50.1045768Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe5f6ertm/_remote_module_non_scriptable.py
2022-05-18T04:07:50.1206692Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzl45l98_
2022-05-18T04:07:50.1207410Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6h2q4tnr
2022-05-18T04:07:50.1208119Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzl45l98_/_remote_module_non_scriptable.py
2022-05-18T04:07:50.3348489Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6h2q4tnr/_remote_module_non_scriptable.py
2022-05-18T04:07:50.3349204Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:07:50.3702283Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:07:50.3840684Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:07:50.3842544Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:07:50.8861642Z ok (1.875s)
2022-05-18T04:07:50.8861927Z 
2022-05-18T04:07:50.8862477Z ----------------------------------------------------------------------
2022-05-18T04:07:50.8862753Z Ran 1 test in 1.875s
2022-05-18T04:07:50.8864407Z 
2022-05-18T04:07:50.8864468Z OK
2022-05-18T04:07:50.8864559Z 
2022-05-18T04:07:50.8864654Z Generating XML reports...
2022-05-18T04:07:50.8899325Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040749.xml
2022-05-18T04:07:51.7528026Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd352johz
2022-05-18T04:07:51.7528723Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd352johz/_remote_module_non_scriptable.py
2022-05-18T04:07:52.0220512Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:07:52.0230964Z 
2022-05-18T04:07:52.0231095Z Running tests...
2022-05-18T04:07:52.0231542Z ----------------------------------------------------------------------
2022-05-18T04:07:52.3697426Z   test_self_remote_rref_as_self_rpc_arg (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12671
2022-05-18T04:07:52.3722726Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12672
2022-05-18T04:07:52.3747752Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12673
2022-05-18T04:07:52.3773840Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 12674
2022-05-18T04:07:53.0749791Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp69nx5qri
2022-05-18T04:07:53.0750628Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp69nx5qri/_remote_module_non_scriptable.py
2022-05-18T04:07:53.1078800Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx5ucal1f
2022-05-18T04:07:53.1079592Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9d0ayed4
2022-05-18T04:07:53.1080316Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx5ucal1f/_remote_module_non_scriptable.py
2022-05-18T04:07:53.1081001Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9d0ayed4/_remote_module_non_scriptable.py
2022-05-18T04:07:53.1387582Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzxmyxcb3
2022-05-18T04:07:53.1388367Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzxmyxcb3/_remote_module_non_scriptable.py
2022-05-18T04:07:53.3400293Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:07:53.3713138Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:07:53.3713550Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:07:53.4017595Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:07:53.8816698Z ok (1.858s)
2022-05-18T04:07:53.8816972Z 
2022-05-18T04:07:53.8817394Z ----------------------------------------------------------------------
2022-05-18T04:07:53.8817662Z Ran 1 test in 1.858s
2022-05-18T04:07:53.8817788Z 
2022-05-18T04:07:53.8817852Z OK
2022-05-18T04:07:53.8817943Z 
2022-05-18T04:07:53.8818037Z Generating XML reports...
2022-05-18T04:07:53.8852280Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040752.xml
2022-05-18T04:07:54.7485860Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmbvzhks1
2022-05-18T04:07:54.7486617Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmbvzhks1/_remote_module_non_scriptable.py
2022-05-18T04:07:55.0195412Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:07:55.0205652Z 
2022-05-18T04:07:55.0205923Z Running tests...
2022-05-18T04:07:55.0206353Z ----------------------------------------------------------------------
2022-05-18T04:07:55.3693865Z   test_send_to_rank (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12890
2022-05-18T04:07:55.3720927Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12891
2022-05-18T04:07:55.3746373Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12892
2022-05-18T04:07:55.3772952Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 12893
2022-05-18T04:07:55.9793633Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph46n150w
2022-05-18T04:07:55.9794813Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph46n150w/_remote_module_non_scriptable.py
2022-05-18T04:07:56.0049317Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphc3duu94
2022-05-18T04:07:56.0050208Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphc3duu94/_remote_module_non_scriptable.py
2022-05-18T04:07:56.0257604Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu9paud1i
2022-05-18T04:07:56.0258391Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu9paud1i/_remote_module_non_scriptable.py
2022-05-18T04:07:56.0369472Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsodacjiz
2022-05-18T04:07:56.0370298Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsodacjiz/_remote_module_non_scriptable.py
2022-05-18T04:07:56.2399332Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:07:56.2657064Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:07:56.2857975Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:07:56.2994899Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:07:56.7814479Z ok (1.761s)
2022-05-18T04:07:56.7814624Z 
2022-05-18T04:07:56.7814948Z ----------------------------------------------------------------------
2022-05-18T04:07:56.7815216Z Ran 1 test in 1.761s
2022-05-18T04:07:56.7815352Z 
2022-05-18T04:07:56.7815416Z OK
2022-05-18T04:07:56.7815509Z 
2022-05-18T04:07:56.7815590Z Generating XML reports...
2022-05-18T04:07:56.7854233Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040755.xml
2022-05-18T04:07:57.6564016Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpin2oz6yj
2022-05-18T04:07:57.6565120Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpin2oz6yj/_remote_module_non_scriptable.py
2022-05-18T04:07:57.9267078Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:07:57.9277224Z 
2022-05-18T04:07:57.9277312Z Running tests...
2022-05-18T04:07:57.9277824Z ----------------------------------------------------------------------
2022-05-18T04:07:58.2766668Z   test_server_process_global_profiler (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13109
2022-05-18T04:07:58.2792041Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13110
2022-05-18T04:07:58.2816820Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13111
2022-05-18T04:07:58.2841901Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 13112
2022-05-18T04:07:58.9414063Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxp_5pm65
2022-05-18T04:07:58.9415205Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxp_5pm65/_remote_module_non_scriptable.py
2022-05-18T04:07:58.9541005Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbc039xo5
2022-05-18T04:07:58.9541776Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbc039xo5/_remote_module_non_scriptable.py
2022-05-18T04:07:58.9896456Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi9iz_4cv
2022-05-18T04:07:58.9897540Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi9iz_4cv/_remote_module_non_scriptable.py
2022-05-18T04:07:59.0283012Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj181635y
2022-05-18T04:07:59.0284049Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj181635y/_remote_module_non_scriptable.py
2022-05-18T04:07:59.2025621Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:07:59.2167193Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:07:59.2497551Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:07:59.2883914Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:07:59.7885885Z ok (1.861s)
2022-05-18T04:07:59.7886155Z 
2022-05-18T04:07:59.7886632Z ----------------------------------------------------------------------
2022-05-18T04:07:59.7886893Z Ran 1 test in 1.861s
2022-05-18T04:07:59.7887027Z 
2022-05-18T04:07:59.7887090Z OK
2022-05-18T04:07:59.7887183Z 
2022-05-18T04:07:59.7887279Z Generating XML reports...
2022-05-18T04:07:59.7921445Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040757.xml
2022-05-18T04:08:00.6549085Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp78canmd6
2022-05-18T04:08:00.6550643Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp78canmd6/_remote_module_non_scriptable.py
2022-05-18T04:08:00.9215898Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:08:00.9225669Z 
2022-05-18T04:08:00.9225794Z Running tests...
2022-05-18T04:08:00.9226203Z ----------------------------------------------------------------------
2022-05-18T04:08:01.2630930Z   test_set_and_get_default_rpc_timeout (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13328
2022-05-18T04:08:01.2656785Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13329
2022-05-18T04:08:01.2683556Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13330
2022-05-18T04:08:01.2711693Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 13331
2022-05-18T04:08:01.9437937Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3w7vnjmb
2022-05-18T04:08:01.9438693Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3w7vnjmb/_remote_module_non_scriptable.py
2022-05-18T04:08:01.9556680Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpee0prfb1
2022-05-18T04:08:01.9557435Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpee0prfb1/_remote_module_non_scriptable.py
2022-05-18T04:08:02.0172680Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr8p5ttbl
2022-05-18T04:08:02.0173495Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr8p5ttbl/_remote_module_non_scriptable.py
2022-05-18T04:08:02.0212322Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptlmllefj
2022-05-18T04:08:02.0213363Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptlmllefj/_remote_module_non_scriptable.py
2022-05-18T04:08:02.2005158Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:08:02.2102198Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:08:02.2795828Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:08:02.2822809Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:08:02.7754788Z ok (1.853s)
2022-05-18T04:08:02.7755021Z 
2022-05-18T04:08:02.7755477Z ----------------------------------------------------------------------
2022-05-18T04:08:02.7755768Z Ran 1 test in 1.853s
2022-05-18T04:08:02.7756153Z 
2022-05-18T04:08:02.7756219Z OK
2022-05-18T04:08:02.7756313Z 
2022-05-18T04:08:02.7756409Z Generating XML reports...
2022-05-18T04:08:02.7791353Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040800.xml
2022-05-18T04:08:03.6503994Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprdml6rzx
2022-05-18T04:08:03.6504862Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprdml6rzx/_remote_module_non_scriptable.py
2022-05-18T04:08:03.9206344Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:08:03.9216321Z 
2022-05-18T04:08:03.9216599Z Running tests...
2022-05-18T04:08:03.9217071Z ----------------------------------------------------------------------
2022-05-18T04:08:04.2685297Z   test_shutdown_errors (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13547
2022-05-18T04:08:04.2710976Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13548
2022-05-18T04:08:04.2736105Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13549
2022-05-18T04:08:04.2762428Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 13550
2022-05-18T04:08:04.8747955Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1l7kpmb9
2022-05-18T04:08:04.8748941Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1l7kpmb9/_remote_module_non_scriptable.py
2022-05-18T04:08:04.8979035Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9xjyhpdd
2022-05-18T04:08:04.8979816Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9xjyhpdd/_remote_module_non_scriptable.py
2022-05-18T04:08:04.9361563Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfug3le6k
2022-05-18T04:08:04.9362562Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfug3le6k/_remote_module_non_scriptable.py
2022-05-18T04:08:04.9550787Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdvtsd553
2022-05-18T04:08:04.9551585Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdvtsd553/_remote_module_non_scriptable.py
2022-05-18T04:08:05.1399897Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:08:05.1626394Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:08:05.2025460Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:08:05.2218391Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:08:05.2338319Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:08:05.2338749Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:08:05.2441096Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:08:05.2441688Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T04:08:05.2442409Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:08:05.2443000Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:08:05.2443514Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:08:05.2444656Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:08:05.5108078Z On WorkerInfo(id=2, name=worker2):
2022-05-18T04:08:05.5110543Z RuntimeError('simulation')
2022-05-18T04:08:05.5113424Z Traceback (most recent call last):
2022-05-18T04:08:05.5114518Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function
2022-05-18T04:08:05.5115493Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-05-18T04:08:05.5116506Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 4728, in raise_error
2022-05-18T04:08:05.5117290Z     raise RuntimeError('simulation')
2022-05-18T04:08:05.5117728Z RuntimeError: simulation
2022-05-18T04:08:05.5118186Z On WorkerInfo(id=3, name=worker3):
2022-05-18T04:08:05.5118710Z RuntimeError('simulation')
2022-05-18T04:08:05.5119153Z Traceback (most recent call last):
2022-05-18T04:08:05.5120013Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function
2022-05-18T04:08:05.5120751Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-05-18T04:08:05.5121746Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 4728, in raise_error
2022-05-18T04:08:05.5122521Z     raise RuntimeError('simulation')
2022-05-18T04:08:05.5122987Z RuntimeError: simulation
2022-05-18T04:08:05.5123253Z 
2022-05-18T04:08:05.5123263Z 
2022-05-18T04:08:05.5123893Z [W tensorpipe_agent.cpp:627] RPC agent for worker3 won't send response to request #3 to worker0, as the agent is shutting down
2022-05-18T04:08:05.5124553Z On WorkerInfo(id=1, name=worker1):
2022-05-18T04:08:05.5125058Z RuntimeError('simulation')
2022-05-18T04:08:05.5125514Z Traceback (most recent call last):
2022-05-18T04:08:05.5126369Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function
2022-05-18T04:08:05.5126906Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-05-18T04:08:05.5127626Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 4728, in raise_error
2022-05-18T04:08:05.5128188Z     raise RuntimeError('simulation')
2022-05-18T04:08:05.5128590Z RuntimeError: simulation
2022-05-18T04:08:05.5128768Z 
2022-05-18T04:08:05.5129185Z [W tensorpipe_agent.cpp:728] RPC agent for worker3 encountered error when reading incoming request from worker0: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2022-05-18T04:08:05.5130024Z [W tensorpipe_agent.cpp:942] RPC agent for worker0 encountered error when reading incoming response from worker3: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2022-05-18T04:08:05.5130873Z [W tensorpipe_agent.cpp:728] RPC agent for worker0 encountered error when reading incoming request from worker3: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2022-05-18T04:08:05.5132218Z ERROR:torch.distributed.rpc.api:Failed to respond to 'Shutdown Proceed' in time, got error Followers ['worker3', 'worker2', 'worker1'] timed out in _all_gather after 0.00 seconds. The first exception is eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2022-05-18T04:08:05.5133264Z [W tensorpipe_agent.cpp:728] RPC agent for worker1 encountered error when reading incoming request from worker0: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2022-05-18T04:08:05.5134086Z [W tensorpipe_agent.cpp:728] RPC agent for worker2 encountered error when reading incoming request from worker0: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2022-05-18T04:08:05.5134923Z [W tensorpipe_agent.cpp:728] RPC agent for worker0 encountered error when reading incoming request from worker1: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2022-05-18T04:08:05.5135723Z [W tensorpipe_agent.cpp:728] RPC agent for worker0 encountered error when reading incoming request from worker2: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2022-05-18T04:08:05.7804839Z ok (1.858s)
2022-05-18T04:08:05.7805486Z 
2022-05-18T04:08:05.7805801Z ----------------------------------------------------------------------
2022-05-18T04:08:05.7806073Z Ran 1 test in 1.859s
2022-05-18T04:08:05.7806192Z 
2022-05-18T04:08:05.7806257Z OK
2022-05-18T04:08:05.7806423Z 
2022-05-18T04:08:05.7806522Z Generating XML reports...
2022-05-18T04:08:05.7841656Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040803.xml
2022-05-18T04:08:06.6486843Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3no7yw0z
2022-05-18T04:08:06.6487681Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3no7yw0z/_remote_module_non_scriptable.py
2022-05-18T04:08:06.9225417Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:08:06.9235735Z 
2022-05-18T04:08:06.9235846Z Running tests...
2022-05-18T04:08:06.9236404Z ----------------------------------------------------------------------
2022-05-18T04:08:07.2740629Z   test_shutdown_followed_by_rpc (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13778
2022-05-18T04:08:07.2766091Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13779
2022-05-18T04:08:07.2792559Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13780
2022-05-18T04:08:07.2819486Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 13781
2022-05-18T04:08:07.9933652Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeqmtp7qw
2022-05-18T04:08:07.9934196Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeqmtp7qw/_remote_module_non_scriptable.py
2022-05-18T04:08:08.0056760Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkkrwlhnn
2022-05-18T04:08:08.0057726Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkkrwlhnn/_remote_module_non_scriptable.py
2022-05-18T04:08:08.1064343Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu615uaq9
2022-05-18T04:08:08.1065110Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdaen7q0h
2022-05-18T04:08:08.1065853Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu615uaq9/_remote_module_non_scriptable.py
2022-05-18T04:08:08.1066607Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdaen7q0h/_remote_module_non_scriptable.py
2022-05-18T04:08:08.2578633Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:08:08.2705517Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:08:08.3717772Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:08:08.3718342Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:08:08.8863177Z ok (1.962s)
2022-05-18T04:08:08.8863473Z 
2022-05-18T04:08:08.8863951Z ----------------------------------------------------------------------
2022-05-18T04:08:08.8864188Z Ran 1 test in 1.963s
2022-05-18T04:08:08.8864304Z 
2022-05-18T04:08:08.8864364Z OK
2022-05-18T04:08:08.8864463Z 
2022-05-18T04:08:08.8864555Z Generating XML reports...
2022-05-18T04:08:08.8899152Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040806.xml
2022-05-18T04:08:09.7645285Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5vzaf91w
2022-05-18T04:08:09.7645790Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5vzaf91w/_remote_module_non_scriptable.py
2022-05-18T04:08:10.0378963Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:08:10.0389309Z 
2022-05-18T04:08:10.0389458Z Running tests...
2022-05-18T04:08:10.0389904Z ----------------------------------------------------------------------
2022-05-18T04:08:10.3909731Z   test_stress_heavy_rpc (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13997
2022-05-18T04:08:10.3933741Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13998
2022-05-18T04:08:10.3959320Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13999
2022-05-18T04:08:10.3986559Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 14000
2022-05-18T04:08:11.0600899Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv5jm2xkt
2022-05-18T04:08:11.0601874Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv5jm2xkt/_remote_module_non_scriptable.py
2022-05-18T04:08:11.1292649Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptkop4dgl
2022-05-18T04:08:11.1293441Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptkop4dgl/_remote_module_non_scriptable.py
2022-05-18T04:08:11.1400499Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnoudgd65
2022-05-18T04:08:11.1401302Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnoudgd65/_remote_module_non_scriptable.py
2022-05-18T04:08:11.1531851Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe666j_7o
2022-05-18T04:08:11.1532598Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe666j_7o/_remote_module_non_scriptable.py
2022-05-18T04:08:11.3288914Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:08:11.3930920Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:08:11.4073810Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:08:11.4182380Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:08:11.7123901Z Rank 0 finished testing 20 times in 0.06757402420043945 seconds.
2022-05-18T04:08:11.7398174Z Rank 3 finished testing 20 times in 0.07496142387390137 seconds.
2022-05-18T04:08:11.7408765Z Rank 1 finished testing 20 times in 0.084808349609375 seconds.
2022-05-18T04:08:11.7498715Z Rank 2 finished testing 20 times in 0.08521294593811035 seconds.
2022-05-18T04:08:12.0031055Z ok (1.964s)
2022-05-18T04:08:12.0031308Z 
2022-05-18T04:08:12.0031851Z ----------------------------------------------------------------------
2022-05-18T04:08:12.0032165Z Ran 1 test in 1.964s
2022-05-18T04:08:12.0032281Z 
2022-05-18T04:08:12.0032345Z OK
2022-05-18T04:08:12.0032438Z 
2022-05-18T04:08:12.0032533Z Generating XML reports...
2022-05-18T04:08:12.0067330Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040810.xml
2022-05-18T04:08:12.8822787Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpntrtveor
2022-05-18T04:08:12.8823478Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpntrtveor/_remote_module_non_scriptable.py
2022-05-18T04:08:13.1540682Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:08:13.1550971Z 
2022-05-18T04:08:13.1551059Z Running tests...
2022-05-18T04:08:13.1551558Z ----------------------------------------------------------------------
2022-05-18T04:08:13.4964293Z   test_stress_heavy_rpc_torchscript (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14216
2022-05-18T04:08:13.4991726Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14217
2022-05-18T04:08:13.5018286Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14218
2022-05-18T04:08:13.5046991Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 14219
2022-05-18T04:08:14.1501386Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj8rstxh3
2022-05-18T04:08:14.1502267Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj8rstxh3/_remote_module_non_scriptable.py
2022-05-18T04:08:14.1594836Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptjynw361
2022-05-18T04:08:14.1596058Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptjynw361/_remote_module_non_scriptable.py
2022-05-18T04:08:14.1745500Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3_wq4zcg
2022-05-18T04:08:14.1746239Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3_wq4zcg/_remote_module_non_scriptable.py
2022-05-18T04:08:14.1806407Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe5i_0es7
2022-05-18T04:08:14.1807169Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe5i_0es7/_remote_module_non_scriptable.py
2022-05-18T04:08:14.4106207Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:08:14.4231004Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:08:14.4374541Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:08:14.4449832Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:08:14.7852285Z Rank 0 finished testing 20 times in 0.12448477745056152 seconds.
2022-05-18T04:08:14.8079848Z Rank 1 finished testing 20 times in 0.13738775253295898 seconds.
2022-05-18T04:08:14.8117909Z Rank 3 finished testing 20 times in 0.13113713264465332 seconds.
2022-05-18T04:08:14.8175313Z Rank 2 finished testing 20 times in 0.13686275482177734 seconds.
2022-05-18T04:08:15.1093369Z ok (1.954s)
2022-05-18T04:08:15.1093583Z 
2022-05-18T04:08:15.1094033Z ----------------------------------------------------------------------
2022-05-18T04:08:15.1094432Z Ran 1 test in 1.954s
2022-05-18T04:08:15.1094610Z 
2022-05-18T04:08:15.1094706Z OK
2022-05-18T04:08:15.1094850Z 
2022-05-18T04:08:15.1094985Z Generating XML reports...
2022-05-18T04:08:15.1130983Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040813.xml
2022-05-18T04:08:15.9894261Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyu6f0h59
2022-05-18T04:08:15.9894978Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyu6f0h59/_remote_module_non_scriptable.py
2022-05-18T04:08:16.2629956Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:08:16.2640825Z 
2022-05-18T04:08:16.2641163Z Running tests...
2022-05-18T04:08:16.2641768Z ----------------------------------------------------------------------
2022-05-18T04:08:16.6118442Z   test_stress_light_rpc (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14435
2022-05-18T04:08:16.6142787Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14436
2022-05-18T04:08:16.6168916Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14437
2022-05-18T04:08:16.6194596Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 14438
2022-05-18T04:08:17.2758244Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppruub7wa
2022-05-18T04:08:17.2759072Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppruub7wa/_remote_module_non_scriptable.py
2022-05-18T04:08:17.2806689Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph_ojdpac
2022-05-18T04:08:17.2807483Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph_ojdpac/_remote_module_non_scriptable.py
2022-05-18T04:08:17.3242759Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwitb7_8_
2022-05-18T04:08:17.3243526Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwitb7_8_/_remote_module_non_scriptable.py
2022-05-18T04:08:17.3394886Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzs_aikm0
2022-05-18T04:08:17.3395920Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzs_aikm0/_remote_module_non_scriptable.py
2022-05-18T04:08:17.5408226Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:08:17.5475074Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:08:17.5889336Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:08:17.6013732Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:08:18.1073265Z Rank 0 finished testing 1000 times in 0.2666280269622803 seconds.
2022-05-18T04:08:18.1427777Z Rank 1 finished testing 1000 times in 0.28224992752075195 seconds.
2022-05-18T04:08:18.1434884Z Rank 2 finished testing 1000 times in 0.28319287300109863 seconds.
2022-05-18T04:08:18.1863411Z Rank 3 finished testing 1000 times in 0.3256988525390625 seconds.
2022-05-18T04:08:18.4242217Z ok (2.160s)
2022-05-18T04:08:18.4242414Z 
2022-05-18T04:08:18.4242738Z ----------------------------------------------------------------------
2022-05-18T04:08:18.4243025Z Ran 1 test in 2.160s
2022-05-18T04:08:18.4243140Z 
2022-05-18T04:08:18.4243192Z OK
2022-05-18T04:08:18.4243285Z 
2022-05-18T04:08:18.4243383Z Generating XML reports...
2022-05-18T04:08:18.4278491Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040816.xml
2022-05-18T04:08:19.3138055Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2lrd_3ae
2022-05-18T04:08:19.3138976Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2lrd_3ae/_remote_module_non_scriptable.py
2022-05-18T04:08:19.5859035Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:08:19.5869284Z 
2022-05-18T04:08:19.5869389Z Running tests...
2022-05-18T04:08:19.5870364Z ----------------------------------------------------------------------
2022-05-18T04:08:19.9368451Z   test_use_rpc_pickler (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14654
2022-05-18T04:08:19.9392566Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14655
2022-05-18T04:08:19.9416460Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14656
2022-05-18T04:08:19.9441881Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 14657
2022-05-18T04:08:20.5522589Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbn6244yn
2022-05-18T04:08:20.5523380Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbn6244yn/_remote_module_non_scriptable.py
2022-05-18T04:08:20.5863534Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk48y2_1x
2022-05-18T04:08:20.5864286Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk48y2_1x/_remote_module_non_scriptable.py
2022-05-18T04:08:20.5918989Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpphtzu_er
2022-05-18T04:08:20.5920265Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpphtzu_er/_remote_module_non_scriptable.py
2022-05-18T04:08:20.6029664Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx3mo38wi
2022-05-18T04:08:20.6030442Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx3mo38wi/_remote_module_non_scriptable.py
2022-05-18T04:08:20.8129333Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:08:20.8459918Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:08:20.8528415Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:08:20.8740288Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:08:21.0478138Z ok (1.461s)
2022-05-18T04:08:21.0478478Z 
2022-05-18T04:08:21.0478916Z ----------------------------------------------------------------------
2022-05-18T04:08:21.0479419Z Ran 1 test in 1.461s
2022-05-18T04:08:21.0479539Z 
2022-05-18T04:08:21.0479602Z OK
2022-05-18T04:08:21.0479694Z 
2022-05-18T04:08:21.0479775Z Generating XML reports...
2022-05-18T04:08:21.0513680Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040819.xml
2022-05-18T04:08:21.9015777Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7mn5grxd
2022-05-18T04:08:21.9016399Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7mn5grxd/_remote_module_non_scriptable.py
2022-05-18T04:08:22.1708062Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:08:22.1717746Z 
2022-05-18T04:08:22.1717875Z Running tests...
2022-05-18T04:08:22.1718488Z ----------------------------------------------------------------------
2022-05-18T04:08:22.5203137Z   test_use_rref_after_shutdown (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14709
2022-05-18T04:08:22.5227671Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14710
2022-05-18T04:08:22.5252309Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14711
2022-05-18T04:08:22.5278683Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 14712
2022-05-18T04:08:23.1547635Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbuvzc3b1
2022-05-18T04:08:23.1548460Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbuvzc3b1/_remote_module_non_scriptable.py
2022-05-18T04:08:23.1615898Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy4s6swob
2022-05-18T04:08:23.1617038Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy4s6swob/_remote_module_non_scriptable.py
2022-05-18T04:08:23.1643636Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp07wltbu2
2022-05-18T04:08:23.1646124Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp07wltbu2/_remote_module_non_scriptable.py
2022-05-18T04:08:23.1875824Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpud45020f
2022-05-18T04:08:23.1877460Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpud45020f/_remote_module_non_scriptable.py
2022-05-18T04:08:23.4151750Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:08:23.4185191Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:08:23.4241599Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:08:23.4464078Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:08:23.9319645Z ok (1.760s)
2022-05-18T04:08:23.9319927Z 
2022-05-18T04:08:23.9320450Z ----------------------------------------------------------------------
2022-05-18T04:08:23.9320728Z Ran 1 test in 1.760s
2022-05-18T04:08:23.9320830Z 
2022-05-18T04:08:23.9320891Z OK
2022-05-18T04:08:23.9320982Z 
2022-05-18T04:08:23.9321076Z Generating XML reports...
2022-05-18T04:08:23.9358018Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040822.xml
2022-05-18T04:08:24.8082674Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk7b60d_d
2022-05-18T04:08:24.8083345Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk7b60d_d/_remote_module_non_scriptable.py
2022-05-18T04:08:25.0764340Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:08:25.0774623Z 
2022-05-18T04:08:25.0774733Z Running tests...
2022-05-18T04:08:25.0775688Z ----------------------------------------------------------------------
2022-05-18T04:08:25.4222192Z   test_user_rref_backward (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14928
2022-05-18T04:08:25.4246942Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14929
2022-05-18T04:08:25.4272015Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14930
2022-05-18T04:08:25.4298127Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 14931
2022-05-18T04:08:26.0303312Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2um8yz3f
2022-05-18T04:08:26.0304044Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2um8yz3f/_remote_module_non_scriptable.py
2022-05-18T04:08:26.0747980Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy10mcd80
2022-05-18T04:08:26.0748729Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy10mcd80/_remote_module_non_scriptable.py
2022-05-18T04:08:26.0947302Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqzfpfmlh
2022-05-18T04:08:26.0948426Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqzfpfmlh/_remote_module_non_scriptable.py
2022-05-18T04:08:26.1004441Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp72wup0zv
2022-05-18T04:08:26.1005754Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp72wup0zv/_remote_module_non_scriptable.py
2022-05-18T04:08:26.2932880Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:08:26.3377180Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:08:26.3597634Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:08:26.3619384Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:08:26.9340711Z ok (1.856s)
2022-05-18T04:08:26.9340973Z 
2022-05-18T04:08:26.9341491Z ----------------------------------------------------------------------
2022-05-18T04:08:26.9341944Z Ran 1 test in 1.856s
2022-05-18T04:08:26.9342156Z 
2022-05-18T04:08:26.9342281Z OK
2022-05-18T04:08:26.9342450Z 
2022-05-18T04:08:26.9342623Z Generating XML reports...
2022-05-18T04:08:26.9379111Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040825.xml
2022-05-18T04:08:27.8065561Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt8ulfpo6
2022-05-18T04:08:27.8066610Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt8ulfpo6/_remote_module_non_scriptable.py
2022-05-18T04:08:28.0797591Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:08:28.0807790Z 
2022-05-18T04:08:28.0807892Z Running tests...
2022-05-18T04:08:28.0808541Z ----------------------------------------------------------------------
2022-05-18T04:08:28.4334623Z   test_user_rrefs_confirmed (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15167
2022-05-18T04:08:28.4359506Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15168
2022-05-18T04:08:28.4384551Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15169
2022-05-18T04:08:28.4410896Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 15170
2022-05-18T04:08:29.1199120Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplmc5azne
2022-05-18T04:08:29.1199902Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdarxsin0
2022-05-18T04:08:29.1200587Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplmc5azne/_remote_module_non_scriptable.py
2022-05-18T04:08:29.1202339Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdarxsin0/_remote_module_non_scriptable.py
2022-05-18T04:08:29.1407117Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkfby4_n8
2022-05-18T04:08:29.1408123Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkfby4_n8/_remote_module_non_scriptable.py
2022-05-18T04:08:29.1837530Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzxprti_f
2022-05-18T04:08:29.1838288Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzxprti_f/_remote_module_non_scriptable.py
2022-05-18T04:08:29.3827532Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:08:29.3846682Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:08:29.4044948Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:08:29.4469588Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:08:30.0456071Z ok (1.964s)
2022-05-18T04:08:30.0456372Z 
2022-05-18T04:08:30.0456871Z ----------------------------------------------------------------------
2022-05-18T04:08:30.0457179Z Ran 1 test in 1.965s
2022-05-18T04:08:30.0457297Z 
2022-05-18T04:08:30.0457360Z OK
2022-05-18T04:08:30.0457460Z 
2022-05-18T04:08:30.0457541Z Generating XML reports...
2022-05-18T04:08:30.0493053Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040828.xml
2022-05-18T04:08:30.9175990Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpchbrxqug
2022-05-18T04:08:30.9177033Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpchbrxqug/_remote_module_non_scriptable.py
2022-05-18T04:08:31.1880841Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:08:31.1889947Z 
2022-05-18T04:08:31.1890065Z Running tests...
2022-05-18T04:08:31.1890523Z ----------------------------------------------------------------------
2022-05-18T04:08:31.5373642Z   test_user_rrefs_confirmed_remote (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15386
2022-05-18T04:08:31.5401276Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15387
2022-05-18T04:08:31.5426275Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15388
2022-05-18T04:08:31.5451745Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 15389
2022-05-18T04:08:32.1596105Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp31_wf87v
2022-05-18T04:08:32.1596912Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp31_wf87v/_remote_module_non_scriptable.py
2022-05-18T04:08:32.1905524Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp85ucxb0y
2022-05-18T04:08:32.1906263Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp85ucxb0y/_remote_module_non_scriptable.py
2022-05-18T04:08:32.2007129Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_vrzbqf5
2022-05-18T04:08:32.2008440Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_vrzbqf5/_remote_module_non_scriptable.py
2022-05-18T04:08:32.2016512Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpaa8pw_vo
2022-05-18T04:08:32.2018736Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpaa8pw_vo/_remote_module_non_scriptable.py
2022-05-18T04:08:32.4255926Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:08:32.4575937Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:08:32.4645410Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:08:32.4651229Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:08:32.9494042Z ok (1.760s)
2022-05-18T04:08:32.9494268Z 
2022-05-18T04:08:32.9494824Z ----------------------------------------------------------------------
2022-05-18T04:08:32.9495416Z Ran 1 test in 1.760s
2022-05-18T04:08:32.9495537Z 
2022-05-18T04:08:32.9495602Z OK
2022-05-18T04:08:32.9495696Z 
2022-05-18T04:08:32.9495799Z Generating XML reports...
2022-05-18T04:08:32.9530241Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040831.xml
2022-05-18T04:08:33.8063266Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdjlxtzzv
2022-05-18T04:08:33.8064235Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdjlxtzzv/_remote_module_non_scriptable.py
2022-05-18T04:08:34.0755773Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:08:34.0766441Z 
2022-05-18T04:08:34.0766715Z Running tests...
2022-05-18T04:08:34.0767375Z ----------------------------------------------------------------------
2022-05-18T04:08:34.4231970Z   test_wait_all (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15605
2022-05-18T04:08:34.4256576Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15606
2022-05-18T04:08:34.4281769Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15607
2022-05-18T04:08:34.4309085Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 15608
2022-05-18T04:08:35.0690981Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps39v_55m
2022-05-18T04:08:35.0691751Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps39v_55m/_remote_module_non_scriptable.py
2022-05-18T04:08:35.0815060Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzfa3fk8c
2022-05-18T04:08:35.0815776Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzfa3fk8c/_remote_module_non_scriptable.py
2022-05-18T04:08:35.1227374Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwfcchetj
2022-05-18T04:08:35.1228121Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwfcchetj/_remote_module_non_scriptable.py
2022-05-18T04:08:35.1379434Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkwxhb2am
2022-05-18T04:08:35.1380723Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkwxhb2am/_remote_module_non_scriptable.py
2022-05-18T04:08:35.3311690Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:08:35.3457177Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:08:35.3886749Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:08:35.4032160Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:08:35.8350321Z ok (1.758s)
2022-05-18T04:08:35.8350521Z 
2022-05-18T04:08:35.8351041Z ----------------------------------------------------------------------
2022-05-18T04:08:35.8351479Z Ran 1 test in 1.758s
2022-05-18T04:08:35.8351626Z 
2022-05-18T04:08:35.8351689Z OK
2022-05-18T04:08:35.8351782Z 
2022-05-18T04:08:35.8351883Z Generating XML reports...
2022-05-18T04:08:35.8391627Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040834.xml
2022-05-18T04:08:36.6958299Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeeasasu8
2022-05-18T04:08:36.6959187Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeeasasu8/_remote_module_non_scriptable.py
2022-05-18T04:08:36.9661402Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:08:36.9671880Z 
2022-05-18T04:08:36.9672208Z Running tests...
2022-05-18T04:08:36.9672828Z ----------------------------------------------------------------------
2022-05-18T04:08:37.3158641Z   test_wait_all_exit_early_builtin (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15824
2022-05-18T04:08:37.3185159Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15825
2022-05-18T04:08:37.3211099Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15826
2022-05-18T04:08:37.3238072Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 15827
2022-05-18T04:08:37.9902720Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxzrtvhfm
2022-05-18T04:08:37.9903691Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxzrtvhfm/_remote_module_non_scriptable.py
2022-05-18T04:08:38.0149431Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp5v2w949
2022-05-18T04:08:38.0150133Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp5v2w949/_remote_module_non_scriptable.py
2022-05-18T04:08:38.0769225Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpql9dyp8v
2022-05-18T04:08:38.0770027Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1rot4kdo
2022-05-18T04:08:38.0772010Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpql9dyp8v/_remote_module_non_scriptable.py
2022-05-18T04:08:38.0772736Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1rot4kdo/_remote_module_non_scriptable.py
2022-05-18T04:08:38.2532063Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:08:38.2775474Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:08:38.3387906Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:08:38.3389060Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:08:38.5994038Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:08:38.6101376Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:08:38.6102241Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:08:38.6103715Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:08:38.6104908Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:08:38.6105793Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T04:08:38.6106872Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:08:38.6200010Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:08:38.9284388Z ok (1.961s)
2022-05-18T04:08:38.9284652Z 
2022-05-18T04:08:38.9285070Z ----------------------------------------------------------------------
2022-05-18T04:08:38.9285334Z Ran 1 test in 1.961s
2022-05-18T04:08:38.9285451Z 
2022-05-18T04:08:38.9285513Z OK
2022-05-18T04:08:38.9285605Z 
2022-05-18T04:08:38.9285703Z Generating XML reports...
2022-05-18T04:08:38.9322419Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040836.xml
2022-05-18T04:08:39.8073042Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf3a5sjjs
2022-05-18T04:08:39.8073837Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf3a5sjjs/_remote_module_non_scriptable.py
2022-05-18T04:08:40.0780913Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:08:40.0791548Z 
2022-05-18T04:08:40.0791675Z Running tests...
2022-05-18T04:08:40.0792089Z ----------------------------------------------------------------------
2022-05-18T04:08:40.4253841Z   test_wait_all_exit_early_python (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16055
2022-05-18T04:08:40.4278787Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16056
2022-05-18T04:08:40.4303580Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16057
2022-05-18T04:08:40.4329527Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 16058
2022-05-18T04:08:41.0819803Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwyswmdud
2022-05-18T04:08:41.0820549Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwyswmdud/_remote_module_non_scriptable.py
2022-05-18T04:08:41.0858376Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt5iri34v
2022-05-18T04:08:41.0859303Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt5iri34v/_remote_module_non_scriptable.py
2022-05-18T04:08:41.1056810Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp00o5e2st
2022-05-18T04:08:41.1057629Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp00o5e2st/_remote_module_non_scriptable.py
2022-05-18T04:08:41.1174546Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptszhv5_4
2022-05-18T04:08:41.1175718Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptszhv5_4/_remote_module_non_scriptable.py
2022-05-18T04:08:41.3476221Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:08:41.3493320Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:08:41.3690370Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:08:41.3787350Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:08:41.6017235Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:08:41.6117287Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T04:08:41.6120835Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:08:41.6122588Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:08:41.6123515Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:08:41.6124608Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:08:41.6125777Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:08:41.6126964Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:08:41.6245439Z On WorkerInfo(id=0, name=worker0):
2022-05-18T04:08:41.6252438Z ValueError('Expected error')
2022-05-18T04:08:41.6252969Z Traceback (most recent call last):
2022-05-18T04:08:41.6253753Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function
2022-05-18T04:08:41.6254315Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-05-18T04:08:41.6255086Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 396, in raise_func
2022-05-18T04:08:41.6255624Z     raise ValueError(expected_err)
2022-05-18T04:08:41.6255977Z ValueError: Expected error
2022-05-18T04:08:41.6256355Z On WorkerInfo(id=1, name=worker1):
2022-05-18T04:08:41.6256774Z ValueError('Expected error')
2022-05-18T04:08:41.6257459Z Traceback (most recent call last):
2022-05-18T04:08:41.6258111Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function
2022-05-18T04:08:41.6258782Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-05-18T04:08:41.6259579Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 396, in raise_func
2022-05-18T04:08:41.6260080Z     raise ValueError(expected_err)
2022-05-18T04:08:41.6260456Z ValueError: Expected error
2022-05-18T04:08:41.6260678Z 
2022-05-18T04:08:41.6260846Z On WorkerInfo(id=0, name=worker0):
2022-05-18T04:08:41.6261252Z ValueError('Expected error')
2022-05-18T04:08:41.6261628Z Traceback (most recent call last):
2022-05-18T04:08:41.6262265Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function
2022-05-18T04:08:41.6262830Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-05-18T04:08:41.6263685Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 396, in raise_func
2022-05-18T04:08:41.6264205Z     raise ValueError(expected_err)
2022-05-18T04:08:41.6264562Z ValueError: Expected error
2022-05-18T04:08:41.6264770Z 
2022-05-18T04:08:41.6264778Z 
2022-05-18T04:08:41.6264916Z On WorkerInfo(id=1, name=worker1):
2022-05-18T04:08:41.6265325Z ValueError('Expected error')
2022-05-18T04:08:41.6265681Z Traceback (most recent call last):
2022-05-18T04:08:41.6266346Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function
2022-05-18T04:08:41.6266956Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-05-18T04:08:41.6267721Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 396, in raise_func
2022-05-18T04:08:41.6268249Z     raise ValueError(expected_err)
2022-05-18T04:08:41.6268598Z ValueError: Expected error
2022-05-18T04:08:41.6268825Z 
2022-05-18T04:08:41.6334951Z On WorkerInfo(id=3, name=worker3):
2022-05-18T04:08:41.6341278Z ValueError('Expected error')
2022-05-18T04:08:41.6341826Z Traceback (most recent call last):
2022-05-18T04:08:41.6342770Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function
2022-05-18T04:08:41.6343724Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-05-18T04:08:41.6344720Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 396, in raise_func
2022-05-18T04:08:41.6345388Z     raise ValueError(expected_err)
2022-05-18T04:08:41.6345858Z ValueError: Expected error
2022-05-18T04:08:41.6346129Z 
2022-05-18T04:08:41.6346325Z On WorkerInfo(id=3, name=worker3):
2022-05-18T04:08:41.6346856Z ValueError('Expected error')
2022-05-18T04:08:41.6347293Z Traceback (most recent call last):
2022-05-18T04:08:41.6348158Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function
2022-05-18T04:08:41.6348920Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-05-18T04:08:41.6349901Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 396, in raise_func
2022-05-18T04:08:41.6350560Z     raise ValueError(expected_err)
2022-05-18T04:08:41.6351002Z ValueError: Expected error
2022-05-18T04:08:41.6351263Z 
2022-05-18T04:08:41.6368339Z On WorkerInfo(id=2, name=worker2):
2022-05-18T04:08:41.6374790Z ValueError('Expected error')
2022-05-18T04:08:41.6375236Z Traceback (most recent call last):
2022-05-18T04:08:41.6375862Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function
2022-05-18T04:08:41.6376346Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-05-18T04:08:41.6376997Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 396, in raise_func
2022-05-18T04:08:41.6377765Z     raise ValueError(expected_err)
2022-05-18T04:08:41.6377968Z ValueError: Expected error
2022-05-18T04:08:41.6378095Z 
2022-05-18T04:08:41.6378287Z On WorkerInfo(id=2, name=worker2):
2022-05-18T04:08:41.6378670Z ValueError('Expected error')
2022-05-18T04:08:41.6378978Z Traceback (most recent call last):
2022-05-18T04:08:41.6379645Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function
2022-05-18T04:08:41.6380172Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-05-18T04:08:41.6380842Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 396, in raise_func
2022-05-18T04:08:41.6381324Z     raise ValueError(expected_err)
2022-05-18T04:08:41.6381636Z ValueError: Expected error
2022-05-18T04:08:41.6381828Z 
2022-05-18T04:08:41.9374056Z ok (1.858s)
2022-05-18T04:08:41.9374303Z 
2022-05-18T04:08:41.9374816Z ----------------------------------------------------------------------
2022-05-18T04:08:41.9375218Z Ran 1 test in 1.858s
2022-05-18T04:08:41.9375335Z 
2022-05-18T04:08:41.9375398Z OK
2022-05-18T04:08:41.9375490Z 
2022-05-18T04:08:41.9375596Z Generating XML reports...
2022-05-18T04:08:41.9409763Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040840.xml
2022-05-18T04:08:42.8126982Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7zgew6bq
2022-05-18T04:08:42.8127657Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7zgew6bq/_remote_module_non_scriptable.py
2022-05-18T04:08:43.0843428Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:08:43.0853386Z 
2022-05-18T04:08:43.0853492Z Running tests...
2022-05-18T04:08:43.0854062Z ----------------------------------------------------------------------
2022-05-18T04:08:43.4294842Z   test_wait_all_exit_early_script_function (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16286
2022-05-18T04:08:43.4317728Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16287
2022-05-18T04:08:43.4341604Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16288
2022-05-18T04:08:43.4367047Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 16289
2022-05-18T04:08:44.0516775Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkd3vlszz
2022-05-18T04:08:44.0517573Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkd3vlszz/_remote_module_non_scriptable.py
2022-05-18T04:08:44.0675003Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzou0du2o
2022-05-18T04:08:44.0675791Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzou0du2o/_remote_module_non_scriptable.py
2022-05-18T04:08:44.0759685Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1m0z5j7g
2022-05-18T04:08:44.0760529Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1m0z5j7g/_remote_module_non_scriptable.py
2022-05-18T04:08:44.0865460Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphjwtkuep
2022-05-18T04:08:44.0866125Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphjwtkuep/_remote_module_non_scriptable.py
2022-05-18T04:08:44.3171117Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:08:44.3309487Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:08:44.3407370Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:08:44.3506890Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:08:44.6338710Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:08:44.6540774Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:08:44.6541960Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:08:44.6542773Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T04:08:44.6544231Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:08:44.6545398Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:08:44.6546522Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:08:44.6550048Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:08:45.0413332Z ok (1.956s)
2022-05-18T04:08:45.0413576Z 
2022-05-18T04:08:45.0414124Z ----------------------------------------------------------------------
2022-05-18T04:08:45.0414560Z Ran 1 test in 1.956s
2022-05-18T04:08:45.0414792Z 
2022-05-18T04:08:45.0414908Z OK
2022-05-18T04:08:45.0415037Z 
2022-05-18T04:08:45.0415131Z Generating XML reports...
2022-05-18T04:08:45.0449902Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040843.xml
2022-05-18T04:08:45.9191228Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqv297cpn
2022-05-18T04:08:45.9191708Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqv297cpn/_remote_module_non_scriptable.py
2022-05-18T04:08:46.1882995Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:08:46.1893032Z 
2022-05-18T04:08:46.1893515Z Running tests...
2022-05-18T04:08:46.1894113Z ----------------------------------------------------------------------
2022-05-18T04:08:46.5377761Z   test_wait_all_multiple_call (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16517
2022-05-18T04:08:46.5402345Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16518
2022-05-18T04:08:46.5427154Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16519
2022-05-18T04:08:46.5453412Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 16520
2022-05-18T04:08:47.2598278Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgxxh29ae
2022-05-18T04:08:47.2599052Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgxxh29ae/_remote_module_non_scriptable.py
2022-05-18T04:08:47.2734059Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqii5vjti
2022-05-18T04:08:47.2735149Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqii5vjti/_remote_module_non_scriptable.py
2022-05-18T04:08:47.2893689Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbkz74yje
2022-05-18T04:08:47.2894508Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbkz74yje/_remote_module_non_scriptable.py
2022-05-18T04:08:47.3045793Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf70y_rnd
2022-05-18T04:08:47.3046553Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf70y_rnd/_remote_module_non_scriptable.py
2022-05-18T04:08:47.5244301Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:08:47.5346097Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:08:47.5581559Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:08:47.5685245Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:08:48.1497744Z ok (1.960s)
2022-05-18T04:08:48.1497957Z 
2022-05-18T04:08:48.1498370Z ----------------------------------------------------------------------
2022-05-18T04:08:48.1498685Z Ran 1 test in 1.960s
2022-05-18T04:08:48.1499087Z 
2022-05-18T04:08:48.1499141Z OK
2022-05-18T04:08:48.1499237Z 
2022-05-18T04:08:48.1499336Z Generating XML reports...
2022-05-18T04:08:48.1535028Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040846.xml
2022-05-18T04:08:49.0248386Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppo02jqvj
2022-05-18T04:08:49.0249393Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppo02jqvj/_remote_module_non_scriptable.py
2022-05-18T04:08:49.2910658Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:08:49.2920285Z 
2022-05-18T04:08:49.2920712Z Running tests...
2022-05-18T04:08:49.2921165Z ----------------------------------------------------------------------
2022-05-18T04:08:49.6399146Z   test_wait_all_raise_in_body (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16736
2022-05-18T04:08:49.6423991Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16737
2022-05-18T04:08:49.6448882Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16738
2022-05-18T04:08:49.6476221Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 16739
2022-05-18T04:08:50.3642765Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsnmgiq_p
2022-05-18T04:08:50.3643491Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsnmgiq_p/_remote_module_non_scriptable.py
2022-05-18T04:08:50.3945420Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpixbv1ti7
2022-05-18T04:08:50.3946149Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpixbv1ti7/_remote_module_non_scriptable.py
2022-05-18T04:08:50.4065390Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpylt7tsei
2022-05-18T04:08:50.4066166Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpylt7tsei/_remote_module_non_scriptable.py
2022-05-18T04:08:50.4069291Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp45_py2gx
2022-05-18T04:08:50.4071613Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp45_py2gx/_remote_module_non_scriptable.py
2022-05-18T04:08:50.6277509Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:08:50.6527194Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:08:50.6649200Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:08:50.6665390Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:08:51.1518178Z ok (1.859s)
2022-05-18T04:08:51.1518393Z 
2022-05-18T04:08:51.1518759Z ----------------------------------------------------------------------
2022-05-18T04:08:51.1519037Z Ran 1 test in 1.860s
2022-05-18T04:08:51.1519152Z 
2022-05-18T04:08:51.1519218Z OK
2022-05-18T04:08:51.1519313Z 
2022-05-18T04:08:51.1519414Z Generating XML reports...
2022-05-18T04:08:51.1554219Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040849.xml
2022-05-18T04:08:52.0121135Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8f08fw6s
2022-05-18T04:08:52.0121732Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8f08fw6s/_remote_module_non_scriptable.py
2022-05-18T04:08:52.2805659Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:08:52.2816648Z 
2022-05-18T04:08:52.2816781Z Running tests...
2022-05-18T04:08:52.2817267Z ----------------------------------------------------------------------
2022-05-18T04:08:52.6332705Z   test_wait_all_raise_in_user_func (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16955
2022-05-18T04:08:52.6358052Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16956
2022-05-18T04:08:52.6382815Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16957
2022-05-18T04:08:52.6408815Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 16958
2022-05-18T04:08:53.3020625Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_ry2tie2
2022-05-18T04:08:53.3021384Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_ry2tie2/_remote_module_non_scriptable.py
2022-05-18T04:08:53.3233599Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkawi7qxp
2022-05-18T04:08:53.3234495Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkawi7qxp/_remote_module_non_scriptable.py
2022-05-18T04:08:53.3462599Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4dhofion
2022-05-18T04:08:53.3463512Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4dhofion/_remote_module_non_scriptable.py
2022-05-18T04:08:53.3464104Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsah2lnc1
2022-05-18T04:08:53.3465211Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsah2lnc1/_remote_module_non_scriptable.py
2022-05-18T04:08:53.5648651Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:08:53.5830118Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:08:53.6077242Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:08:53.6077636Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:08:53.8095206Z On WorkerInfo(id=1, name=worker1):
2022-05-18T04:08:53.8095776Z ValueError('Expected error')
2022-05-18T04:08:53.8096091Z Traceback (most recent call last):
2022-05-18T04:08:53.8096753Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function
2022-05-18T04:08:53.8097286Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-05-18T04:08:53.8098005Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 396, in raise_func
2022-05-18T04:08:53.8098554Z     raise ValueError(expected_err)
2022-05-18T04:08:53.8098870Z ValueError: Expected error
2022-05-18T04:08:53.8099059Z 
2022-05-18T04:08:53.8294275Z On WorkerInfo(id=0, name=worker0):
2022-05-18T04:08:53.8294898Z ValueError('Expected error')
2022-05-18T04:08:53.8295278Z Traceback (most recent call last):
2022-05-18T04:08:53.8296000Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function
2022-05-18T04:08:53.8296597Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-05-18T04:08:53.8297372Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 396, in raise_func
2022-05-18T04:08:53.8297891Z     raise ValueError(expected_err)
2022-05-18T04:08:53.8298255Z ValueError: Expected error
2022-05-18T04:08:53.8298443Z 
2022-05-18T04:08:53.8381565Z On WorkerInfo(id=3, name=worker3):
2022-05-18T04:08:53.8382300Z ValueError('Expected error')
2022-05-18T04:08:53.8382762Z Traceback (most recent call last):
2022-05-18T04:08:53.8383866Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function
2022-05-18T04:08:53.8384568Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-05-18T04:08:53.8385442Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 396, in raise_func
2022-05-18T04:08:53.8386110Z     raise ValueError(expected_err)
2022-05-18T04:08:53.8386894Z ValueError: Expected error
2022-05-18T04:08:53.8387168Z 
2022-05-18T04:08:53.8408044Z On WorkerInfo(id=2, name=worker2):
2022-05-18T04:08:53.8409054Z ValueError('Expected error')
2022-05-18T04:08:53.8409504Z Traceback (most recent call last):
2022-05-18T04:08:53.8410410Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function
2022-05-18T04:08:53.8411158Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-05-18T04:08:53.8412153Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 396, in raise_func
2022-05-18T04:08:53.8412804Z     raise ValueError(expected_err)
2022-05-18T04:08:53.8413254Z ValueError: Expected error
2022-05-18T04:08:53.8413525Z 
2022-05-18T04:08:54.0449336Z ok (1.763s)
2022-05-18T04:08:54.0449499Z 
2022-05-18T04:08:54.0449858Z ----------------------------------------------------------------------
2022-05-18T04:08:54.0450144Z Ran 1 test in 1.763s
2022-05-18T04:08:54.0450284Z 
2022-05-18T04:08:54.0450376Z OK
2022-05-18T04:08:54.0450470Z 
2022-05-18T04:08:54.0450564Z Generating XML reports...
2022-05-18T04:08:54.0484685Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040852.xml
2022-05-18T04:08:54.8763131Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_uf7qxdl
2022-05-18T04:08:54.8763640Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_uf7qxdl/_remote_module_non_scriptable.py
2022-05-18T04:08:55.1462620Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:08:55.1472756Z 
2022-05-18T04:08:55.1472863Z Running tests...
2022-05-18T04:08:55.1473255Z ----------------------------------------------------------------------
2022-05-18T04:08:55.4973692Z   test_wait_all_timeout (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17174
2022-05-18T04:08:55.4996630Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17175
2022-05-18T04:08:55.5020931Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17176
2022-05-18T04:08:55.5046292Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 17177
2022-05-18T04:08:56.1344111Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuswp5fb6
2022-05-18T04:08:56.1345354Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuswp5fb6/_remote_module_non_scriptable.py
2022-05-18T04:08:56.1782657Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplhdcaqih
2022-05-18T04:08:56.1783892Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplhdcaqih/_remote_module_non_scriptable.py
2022-05-18T04:08:56.1848046Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx4zwnag7
2022-05-18T04:08:56.1849014Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx4zwnag7/_remote_module_non_scriptable.py
2022-05-18T04:08:56.1950263Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp92nd24ou
2022-05-18T04:08:56.1951046Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp92nd24ou/_remote_module_non_scriptable.py
2022-05-18T04:08:56.3962226Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:08:56.4403859Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:08:56.4436722Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:08:56.4577980Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:08:56.8248455Z [W tensorpipe_agent.cpp:942] RPC agent for worker3 encountered error when reading incoming response from worker0: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2022-05-18T04:08:56.8341942Z [W tensorpipe_agent.cpp:942] RPC agent for worker1 encountered error when reading incoming response from worker2: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2022-05-18T04:08:57.6824476Z [W tensorpipe_agent.cpp:627] RPC agent for worker1 won't send response to request #3 to worker0, as the agent is shutting down
2022-05-18T04:08:57.7025707Z [W tensorpipe_agent.cpp:627] RPC agent for worker0 won't send response to request #1 to worker3, as the agent is shutting down
2022-05-18T04:08:57.7110061Z [W tensorpipe_agent.cpp:627] RPC agent for worker3 won't send response to request #1 to worker2, as the agent is shutting down
2022-05-18T04:08:57.7135560Z [W tensorpipe_agent.cpp:627] RPC agent for worker2 won't send response to request #1 to worker1, as the agent is shutting down
2022-05-18T04:08:57.9103998Z ok (2.763s)
2022-05-18T04:08:57.9104241Z 
2022-05-18T04:08:57.9104795Z ----------------------------------------------------------------------
2022-05-18T04:08:57.9105142Z Ran 1 test in 2.763s
2022-05-18T04:08:57.9105257Z 
2022-05-18T04:08:57.9105304Z OK
2022-05-18T04:08:57.9105402Z 
2022-05-18T04:08:57.9105498Z Generating XML reports...
2022-05-18T04:08:57.9139512Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040855.xml
2022-05-18T04:08:58.7864531Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8qz76m74
2022-05-18T04:08:58.7865555Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8qz76m74/_remote_module_non_scriptable.py
2022-05-18T04:08:59.0594670Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:08:59.0608734Z 
2022-05-18T04:08:59.0609142Z Running tests...
2022-05-18T04:08:59.0609571Z ----------------------------------------------------------------------
2022-05-18T04:08:59.4092220Z   test_wait_all_with_exception (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17393
2022-05-18T04:08:59.4116261Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17394
2022-05-18T04:08:59.4140446Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17395
2022-05-18T04:08:59.4166697Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 17396
2022-05-18T04:09:00.0894061Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgndvswhh
2022-05-18T04:09:00.0941056Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgndvswhh/_remote_module_non_scriptable.py
2022-05-18T04:09:00.1032748Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt83s85ap
2022-05-18T04:09:00.1033746Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt83s85ap/_remote_module_non_scriptable.py
2022-05-18T04:09:00.1268307Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphs04iz8s
2022-05-18T04:09:00.1269124Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphs04iz8s/_remote_module_non_scriptable.py
2022-05-18T04:09:00.1714066Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp77388fp_
2022-05-18T04:09:00.1715043Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp77388fp_/_remote_module_non_scriptable.py
2022-05-18T04:09:00.3571255Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:09:00.3674865Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:09:00.3873810Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:09:00.4350789Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:09:00.6788435Z On WorkerInfo(id=1, name=worker1):
2022-05-18T04:09:00.6788968Z ValueError('Expected error')
2022-05-18T04:09:00.6789669Z Traceback (most recent call last):
2022-05-18T04:09:00.6790381Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function
2022-05-18T04:09:00.6791117Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-05-18T04:09:00.6791640Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 396, in raise_func
2022-05-18T04:09:00.6791997Z     raise ValueError(expected_err)
2022-05-18T04:09:00.6792195Z ValueError: Expected error
2022-05-18T04:09:00.6799481Z On WorkerInfo(id=1, name=worker1):
2022-05-18T04:09:00.6800450Z ValueError('Expected error')
2022-05-18T04:09:00.6801101Z Traceback (most recent call last):
2022-05-18T04:09:00.6802309Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function
2022-05-18T04:09:00.6803202Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-05-18T04:09:00.6804365Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 396, in raise_func
2022-05-18T04:09:00.6805229Z     raise ValueError(expected_err)
2022-05-18T04:09:00.6805867Z ValueError: Expected error
2022-05-18T04:09:00.6806273Z 
2022-05-18T04:09:00.6806475Z On WorkerInfo(id=1, name=worker1):
2022-05-18T04:09:00.6807169Z ValueError('Expected error')
2022-05-18T04:09:00.6807776Z Traceback (most recent call last):
2022-05-18T04:09:00.6808850Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function
2022-05-18T04:09:00.6809745Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-05-18T04:09:00.6811047Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 396, in raise_func
2022-05-18T04:09:00.6811868Z     raise ValueError(expected_err)
2022-05-18T04:09:00.6812482Z ValueError: Expected error
2022-05-18T04:09:00.6812785Z 
2022-05-18T04:09:00.6812980Z On WorkerInfo(id=1, name=worker1):
2022-05-18T04:09:00.6813676Z ValueError('Expected error')
2022-05-18T04:09:00.6814271Z Traceback (most recent call last):
2022-05-18T04:09:00.6815292Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function
2022-05-18T04:09:00.6816333Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-05-18T04:09:00.6817463Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 396, in raise_func
2022-05-18T04:09:00.6818308Z     raise ValueError(expected_err)
2022-05-18T04:09:00.6818916Z ValueError: Expected error
2022-05-18T04:09:00.6819179Z 
2022-05-18T04:09:00.6819539Z On WorkerInfo(id=1, name=worker1):
2022-05-18T04:09:00.6820200Z ValueError('Expected error')
2022-05-18T04:09:00.6820668Z Traceback (most recent call last):
2022-05-18T04:09:00.6821846Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function
2022-05-18T04:09:00.6822694Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-05-18T04:09:00.6824113Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 396, in raise_func
2022-05-18T04:09:00.6824936Z     raise ValueError(expected_err)
2022-05-18T04:09:00.6825389Z ValueError: Expected error
2022-05-18T04:09:00.6825788Z 
2022-05-18T04:09:00.6825981Z On WorkerInfo(id=1, name=worker1):
2022-05-18T04:09:00.6826505Z ValueError('Expected error')
2022-05-18T04:09:00.6827105Z Traceback (most recent call last):
2022-05-18T04:09:00.6827937Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function
2022-05-18T04:09:00.6828650Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-05-18T04:09:00.6829788Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 396, in raise_func
2022-05-18T04:09:00.6830433Z     raise ValueError(expected_err)
2022-05-18T04:09:00.6831194Z ValueError: Expected error
2022-05-18T04:09:00.6831458Z 
2022-05-18T04:09:00.6831686Z On WorkerInfo(id=1, name=worker1):
2022-05-18T04:09:00.6832206Z ValueError('Expected error')
2022-05-18T04:09:00.6832733Z Traceback (most recent call last):
2022-05-18T04:09:00.6833724Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function
2022-05-18T04:09:00.6834432Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-05-18T04:09:00.6835391Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 396, in raise_func
2022-05-18T04:09:00.6836021Z     raise ValueError(expected_err)
2022-05-18T04:09:00.6836462Z ValueError: Expected error
2022-05-18T04:09:00.6836902Z On WorkerInfo(id=1, name=worker1):
2022-05-18T04:09:00.6837397Z ValueError('Expected error')
2022-05-18T04:09:00.6837843Z Traceback (most recent call last):
2022-05-18T04:09:00.6838669Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function
2022-05-18T04:09:00.6839360Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-05-18T04:09:00.6840322Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 396, in raise_func
2022-05-18T04:09:00.6840984Z     raise ValueError(expected_err)
2022-05-18T04:09:00.6841410Z ValueError: Expected error
2022-05-18T04:09:00.6841670Z 
2022-05-18T04:09:00.6841861Z On WorkerInfo(id=1, name=worker1):
2022-05-18T04:09:00.6842381Z ValueError('Expected error')
2022-05-18T04:09:00.6842826Z Traceback (most recent call last):
2022-05-18T04:09:00.6843637Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function
2022-05-18T04:09:00.6844342Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-05-18T04:09:00.6845306Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 396, in raise_func
2022-05-18T04:09:00.6845958Z     raise ValueError(expected_err)
2022-05-18T04:09:00.6846408Z ValueError: Expected error
2022-05-18T04:09:00.6846668Z 
2022-05-18T04:09:00.6846866Z On WorkerInfo(id=1, name=worker1):
2022-05-18T04:09:00.6847380Z ValueError('Expected error')
2022-05-18T04:09:00.6847814Z Traceback (most recent call last):
2022-05-18T04:09:00.6848710Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function
2022-05-18T04:09:00.6849425Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-05-18T04:09:00.6850387Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 396, in raise_func
2022-05-18T04:09:00.6851040Z     raise ValueError(expected_err)
2022-05-18T04:09:00.6851486Z ValueError: Expected error
2022-05-18T04:09:00.6851745Z 
2022-05-18T04:09:00.6851754Z 
2022-05-18T04:09:00.6851764Z 
2022-05-18T04:09:00.6937999Z On WorkerInfo(id=2, name=worker2):
2022-05-18T04:09:00.6938752Z ValueError('Expected error')
2022-05-18T04:09:00.6939223Z Traceback (most recent call last):
2022-05-18T04:09:00.6940139Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function
2022-05-18T04:09:00.6940854Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-05-18T04:09:00.6941835Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 396, in raise_func
2022-05-18T04:09:00.6942500Z     raise ValueError(expected_err)
2022-05-18T04:09:00.6943145Z ValueError: Expected error
2022-05-18T04:09:00.6943403Z 
2022-05-18T04:09:00.6951342Z On WorkerInfo(id=2, name=worker2):
2022-05-18T04:09:00.6952178Z ValueError('Expected error')
2022-05-18T04:09:00.6953569Z Traceback (most recent call last):
2022-05-18T04:09:00.6954695Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function
2022-05-18T04:09:00.6955999Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-05-18T04:09:00.6956973Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 396, in raise_func
2022-05-18T04:09:00.6958476Z     raise ValueError(expected_err)
2022-05-18T04:09:00.6972640Z ValueError: Expected error
2022-05-18T04:09:00.6972967Z On WorkerInfo(id=2, name=worker2):
2022-05-18T04:09:00.6973419Z ValueError('Expected error')
2022-05-18T04:09:00.6973763Z Traceback (most recent call last):
2022-05-18T04:09:00.6976284Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function
2022-05-18T04:09:00.6978289Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-05-18T04:09:00.6980468Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 396, in raise_func
2022-05-18T04:09:00.6983189Z     raise ValueError(expected_err)
2022-05-18T04:09:00.6983551Z ValueError: Expected error
2022-05-18T04:09:00.6983837Z 
2022-05-18T04:09:00.6984037Z On WorkerInfo(id=2, name=worker2):
2022-05-18T04:09:00.6984652Z ValueError('Expected error')
2022-05-18T04:09:00.6985044Z Traceback (most recent call last):
2022-05-18T04:09:00.6985920Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function
2022-05-18T04:09:00.6986635Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-05-18T04:09:00.6987616Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 396, in raise_func
2022-05-18T04:09:00.6988288Z     raise ValueError(expected_err)
2022-05-18T04:09:00.6988739Z ValueError: Expected error
2022-05-18T04:09:00.6988983Z 
2022-05-18T04:09:00.6989177Z On WorkerInfo(id=2, name=worker2):
2022-05-18T04:09:00.6989701Z ValueError('Expected error')
2022-05-18T04:09:00.6990159Z Traceback (most recent call last):
2022-05-18T04:09:00.6990987Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function
2022-05-18T04:09:00.6991709Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-05-18T04:09:00.6992684Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 396, in raise_func
2022-05-18T04:09:00.6993338Z     raise ValueError(expected_err)
2022-05-18T04:09:00.6993768Z ValueError: Expected error
2022-05-18T04:09:00.6994038Z 
2022-05-18T04:09:00.6994228Z On WorkerInfo(id=2, name=worker2):
2022-05-18T04:09:00.6994740Z ValueError('Expected error')
2022-05-18T04:09:00.6995179Z Traceback (most recent call last):
2022-05-18T04:09:00.6996041Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function
2022-05-18T04:09:00.6996755Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-05-18T04:09:00.6997706Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 396, in raise_func
2022-05-18T04:09:00.6998374Z     raise ValueError(expected_err)
2022-05-18T04:09:00.6998832Z ValueError: Expected error
2022-05-18T04:09:00.6999096Z 
2022-05-18T04:09:00.6999295Z On WorkerInfo(id=2, name=worker2):
2022-05-18T04:09:00.6999795Z ValueError('Expected error')
2022-05-18T04:09:00.7000256Z Traceback (most recent call last):
2022-05-18T04:09:00.7001095Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function
2022-05-18T04:09:00.7001801Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-05-18T04:09:00.7002779Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 396, in raise_func
2022-05-18T04:09:00.7003438Z     raise ValueError(expected_err)
2022-05-18T04:09:00.7003857Z ValueError: Expected error
2022-05-18T04:09:00.7004119Z 
2022-05-18T04:09:00.7004128Z 
2022-05-18T04:09:00.7016812Z On WorkerInfo(id=2, name=worker2):
2022-05-18T04:09:00.7018046Z ValueError('Expected error')
2022-05-18T04:09:00.7022291Z Traceback (most recent call last):
2022-05-18T04:09:00.7023948Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function
2022-05-18T04:09:00.7024542Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-05-18T04:09:00.7025312Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 396, in raise_func
2022-05-18T04:09:00.7025868Z     raise ValueError(expected_err)
2022-05-18T04:09:00.7026217Z ValueError: Expected error
2022-05-18T04:09:00.7026438Z 
2022-05-18T04:09:00.7026607Z On WorkerInfo(id=0, name=worker0):
2022-05-18T04:09:00.7027050Z ValueError('Expected error')
2022-05-18T04:09:00.7027411Z Traceback (most recent call last):
2022-05-18T04:09:00.7028087Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function
2022-05-18T04:09:00.7028658Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-05-18T04:09:00.7029431Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 396, in raise_func
2022-05-18T04:09:00.7030177Z     raise ValueError(expected_err)
2022-05-18T04:09:00.7030565Z ValueError: Expected error
2022-05-18T04:09:00.7030940Z On WorkerInfo(id=2, name=worker2):
2022-05-18T04:09:00.7031362Z ValueError('Expected error')
2022-05-18T04:09:00.7031743Z Traceback (most recent call last):
2022-05-18T04:09:00.7032437Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function
2022-05-18T04:09:00.7032950Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-05-18T04:09:00.7033733Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 396, in raise_func
2022-05-18T04:09:00.7034274Z     raise ValueError(expected_err)
2022-05-18T04:09:00.7034639Z ValueError: Expected error
2022-05-18T04:09:00.7034851Z 
2022-05-18T04:09:00.7035008Z On WorkerInfo(id=2, name=worker2):
2022-05-18T04:09:00.7035435Z ValueError('Expected error')
2022-05-18T04:09:00.7035812Z Traceback (most recent call last):
2022-05-18T04:09:00.7036473Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function
2022-05-18T04:09:00.7037056Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-05-18T04:09:00.7037814Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 396, in raise_func
2022-05-18T04:09:00.7038317Z     raise ValueError(expected_err)
2022-05-18T04:09:00.7038697Z ValueError: Expected error
2022-05-18T04:09:00.7038911Z 
2022-05-18T04:09:00.7038917Z 
2022-05-18T04:09:00.7039106Z On WorkerInfo(id=0, name=worker0):
2022-05-18T04:09:00.7039536Z ValueError('Expected error')
2022-05-18T04:09:00.7039887Z Traceback (most recent call last):
2022-05-18T04:09:00.7040549Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function
2022-05-18T04:09:00.7041133Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-05-18T04:09:00.7041884Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 396, in raise_func
2022-05-18T04:09:00.7042415Z     raise ValueError(expected_err)
2022-05-18T04:09:00.7042791Z ValueError: Expected error
2022-05-18T04:09:00.7043009Z 
2022-05-18T04:09:00.7043183Z On WorkerInfo(id=0, name=worker0):
2022-05-18T04:09:00.7043584Z ValueError('Expected error')
2022-05-18T04:09:00.7043958Z Traceback (most recent call last):
2022-05-18T04:09:00.7044607Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function
2022-05-18T04:09:00.7045155Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-05-18T04:09:00.7045905Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 396, in raise_func
2022-05-18T04:09:00.7046647Z     raise ValueError(expected_err)
2022-05-18T04:09:00.7047025Z ValueError: Expected error
2022-05-18T04:09:00.7047220Z 
2022-05-18T04:09:00.7055039Z On WorkerInfo(id=0, name=worker0):
2022-05-18T04:09:00.7055683Z ValueError('Expected error')
2022-05-18T04:09:00.7056179Z Traceback (most recent call last):
2022-05-18T04:09:00.7057058Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function
2022-05-18T04:09:00.7057769Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-05-18T04:09:00.7058749Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 396, in raise_func
2022-05-18T04:09:00.7059419Z     raise ValueError(expected_err)
2022-05-18T04:09:00.7059857Z ValueError: Expected error
2022-05-18T04:09:00.7060122Z 
2022-05-18T04:09:00.7060315Z On WorkerInfo(id=0, name=worker0):
2022-05-18T04:09:00.7060828Z ValueError('Expected error')
2022-05-18T04:09:00.7061283Z Traceback (most recent call last):
2022-05-18T04:09:00.7062138Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function
2022-05-18T04:09:00.7062861Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-05-18T04:09:00.7064158Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 396, in raise_func
2022-05-18T04:09:00.7064805Z     raise ValueError(expected_err)
2022-05-18T04:09:00.7065263Z ValueError: Expected error
2022-05-18T04:09:00.7065530Z 
2022-05-18T04:09:00.7065726Z On WorkerInfo(id=0, name=worker0):
2022-05-18T04:09:00.7066233Z ValueError('Expected error')
2022-05-18T04:09:00.7066686Z Traceback (most recent call last):
2022-05-18T04:09:00.7067576Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function
2022-05-18T04:09:00.7068301Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-05-18T04:09:00.7069266Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 396, in raise_func
2022-05-18T04:09:00.7069935Z     raise ValueError(expected_err)
2022-05-18T04:09:00.7070394Z ValueError: Expected error
2022-05-18T04:09:00.7070820Z On WorkerInfo(id=0, name=worker0):
2022-05-18T04:09:00.7071337Z ValueError('Expected error')
2022-05-18T04:09:00.7071792Z Traceback (most recent call last):
2022-05-18T04:09:00.7072640Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function
2022-05-18T04:09:00.7073336Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-05-18T04:09:00.7074317Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 396, in raise_func
2022-05-18T04:09:00.7074974Z     raise ValueError(expected_err)
2022-05-18T04:09:00.7075403Z ValueError: Expected error
2022-05-18T04:09:00.7075666Z 
2022-05-18T04:09:00.7075869Z On WorkerInfo(id=0, name=worker0):
2022-05-18T04:09:00.7076391Z ValueError('Expected error')
2022-05-18T04:09:00.7076820Z Traceback (most recent call last):
2022-05-18T04:09:00.7077677Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function
2022-05-18T04:09:00.7078389Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-05-18T04:09:00.7079367Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 396, in raise_func
2022-05-18T04:09:00.7080006Z     raise ValueError(expected_err)
2022-05-18T04:09:00.7080456Z ValueError: Expected error
2022-05-18T04:09:00.7080722Z 
2022-05-18T04:09:00.7080731Z 
2022-05-18T04:09:00.7080922Z On WorkerInfo(id=0, name=worker0):
2022-05-18T04:09:00.7081419Z ValueError('Expected error')
2022-05-18T04:09:00.7081877Z Traceback (most recent call last):
2022-05-18T04:09:00.7082716Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function
2022-05-18T04:09:00.7083651Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-05-18T04:09:00.7084733Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 396, in raise_func
2022-05-18T04:09:00.7085406Z     raise ValueError(expected_err)
2022-05-18T04:09:00.7085856Z ValueError: Expected error
2022-05-18T04:09:00.7086119Z 
2022-05-18T04:09:00.7086290Z On WorkerInfo(id=0, name=worker0):
2022-05-18T04:09:00.7086815Z ValueError('Expected error')
2022-05-18T04:09:00.7087267Z Traceback (most recent call last):
2022-05-18T04:09:00.7088137Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function
2022-05-18T04:09:00.7088863Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-05-18T04:09:00.7089844Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 396, in raise_func
2022-05-18T04:09:00.7090536Z     raise ValueError(expected_err)
2022-05-18T04:09:00.7090966Z ValueError: Expected error
2022-05-18T04:09:00.7091230Z 
2022-05-18T04:09:00.7104126Z On WorkerInfo(id=3, name=worker3):
2022-05-18T04:09:00.7104773Z ValueError('Expected error')
2022-05-18T04:09:00.7106564Z Traceback (most recent call last):
2022-05-18T04:09:00.7107444Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function
2022-05-18T04:09:00.7108372Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-05-18T04:09:00.7109555Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 396, in raise_func
2022-05-18T04:09:00.7110223Z     raise ValueError(expected_err)
2022-05-18T04:09:00.7110774Z ValueError: Expected error
2022-05-18T04:09:00.7111048Z 
2022-05-18T04:09:00.7118972Z On WorkerInfo(id=3, name=worker3):
2022-05-18T04:09:00.7120447Z ValueError('Expected error')
2022-05-18T04:09:00.7120911Z Traceback (most recent call last):
2022-05-18T04:09:00.7121799Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function
2022-05-18T04:09:00.7122544Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-05-18T04:09:00.7123505Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 396, in raise_func
2022-05-18T04:09:00.7124171Z     raise ValueError(expected_err)
2022-05-18T04:09:00.7127181Z ValueError: Expected error
2022-05-18T04:09:00.7130017Z On WorkerInfo(id=3, name=worker3):
2022-05-18T04:09:00.7134163Z ValueError('Expected error')
2022-05-18T04:09:00.7134587Z Traceback (most recent call last):
2022-05-18T04:09:00.7135376Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function
2022-05-18T04:09:00.7135947Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-05-18T04:09:00.7136768Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 396, in raise_func
2022-05-18T04:09:00.7137341Z     raise ValueError(expected_err)
2022-05-18T04:09:00.7137722Z ValueError: Expected error
2022-05-18T04:09:00.7137954Z 
2022-05-18T04:09:00.7138102Z On WorkerInfo(id=3, name=worker3):
2022-05-18T04:09:00.7138552Z ValueError('Expected error')
2022-05-18T04:09:00.7138944Z Traceback (most recent call last):
2022-05-18T04:09:00.7139671Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function
2022-05-18T04:09:00.7140245Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-05-18T04:09:00.7141046Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 396, in raise_func
2022-05-18T04:09:00.7141587Z     raise ValueError(expected_err)
2022-05-18T04:09:00.7141987Z ValueError: Expected error
2022-05-18T04:09:00.7142310Z On WorkerInfo(id=3, name=worker3):
2022-05-18T04:09:00.7142769Z ValueError('Expected error')
2022-05-18T04:09:00.7143533Z Traceback (most recent call last):
2022-05-18T04:09:00.7144330Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function
2022-05-18T04:09:00.7144903Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-05-18T04:09:00.7145716Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 396, in raise_func
2022-05-18T04:09:00.7146268Z     raise ValueError(expected_err)
2022-05-18T04:09:00.7146668Z ValueError: Expected error
2022-05-18T04:09:00.7146902Z 
2022-05-18T04:09:00.7147070Z On WorkerInfo(id=3, name=worker3):
2022-05-18T04:09:00.7147508Z ValueError('Expected error')
2022-05-18T04:09:00.7147859Z Traceback (most recent call last):
2022-05-18T04:09:00.7148585Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function
2022-05-18T04:09:00.7149199Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-05-18T04:09:00.7149977Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 396, in raise_func
2022-05-18T04:09:00.7150548Z     raise ValueError(expected_err)
2022-05-18T04:09:00.7150928Z ValueError: Expected error
2022-05-18T04:09:00.7151151Z 
2022-05-18T04:09:00.7151159Z 
2022-05-18T04:09:00.7151326Z On WorkerInfo(id=3, name=worker3):
2022-05-18T04:09:00.7151737Z ValueError('Expected error')
2022-05-18T04:09:00.7152096Z Traceback (most recent call last):
2022-05-18T04:09:00.7152822Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function
2022-05-18T04:09:00.7153445Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-05-18T04:09:00.7154259Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 396, in raise_func
2022-05-18T04:09:00.7154822Z     raise ValueError(expected_err)
2022-05-18T04:09:00.7155221Z ValueError: Expected error
2022-05-18T04:09:00.7155434Z 
2022-05-18T04:09:00.7155602Z On WorkerInfo(id=3, name=worker3):
2022-05-18T04:09:00.7156026Z ValueError('Expected error')
2022-05-18T04:09:00.7156435Z Traceback (most recent call last):
2022-05-18T04:09:00.7157085Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function
2022-05-18T04:09:00.7157710Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-05-18T04:09:00.7158544Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 396, in raise_func
2022-05-18T04:09:00.7158962Z     raise ValueError(expected_err)
2022-05-18T04:09:00.7159272Z ValueError: Expected error
2022-05-18T04:09:00.7159461Z 
2022-05-18T04:09:00.7159468Z 
2022-05-18T04:09:00.7159597Z On WorkerInfo(id=3, name=worker3):
2022-05-18T04:09:00.7160048Z ValueError('Expected error')
2022-05-18T04:09:00.7160321Z Traceback (most recent call last):
2022-05-18T04:09:00.7160911Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function
2022-05-18T04:09:00.7161519Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-05-18T04:09:00.7162155Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 396, in raise_func
2022-05-18T04:09:00.7162459Z     raise ValueError(expected_err)
2022-05-18T04:09:00.7162671Z ValueError: Expected error
2022-05-18T04:09:00.7162793Z 
2022-05-18T04:09:00.7162883Z On WorkerInfo(id=3, name=worker3):
2022-05-18T04:09:00.7163111Z ValueError('Expected error')
2022-05-18T04:09:00.7163326Z Traceback (most recent call last):
2022-05-18T04:09:00.7163713Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function
2022-05-18T04:09:00.7164034Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-05-18T04:09:00.7164472Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 396, in raise_func
2022-05-18T04:09:00.7164903Z     raise ValueError(expected_err)
2022-05-18T04:09:00.7165114Z ValueError: Expected error
2022-05-18T04:09:00.7165221Z 
2022-05-18T04:09:01.0211747Z ok (1.960s)
2022-05-18T04:09:01.0212016Z 
2022-05-18T04:09:01.0212406Z ----------------------------------------------------------------------
2022-05-18T04:09:01.0212684Z Ran 1 test in 1.960s
2022-05-18T04:09:01.0212803Z 
2022-05-18T04:09:01.0212865Z OK
2022-05-18T04:09:01.0212944Z 
2022-05-18T04:09:01.0213038Z Generating XML reports...
2022-05-18T04:09:01.0246996Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040859.xml
2022-05-18T04:09:01.8932596Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6d1bo0ci
2022-05-18T04:09:01.8933409Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6d1bo0ci/_remote_module_non_scriptable.py
2022-05-18T04:09:02.1603961Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:09:02.1613257Z 
2022-05-18T04:09:02.1613348Z Running tests...
2022-05-18T04:09:02.1613835Z ----------------------------------------------------------------------
2022-05-18T04:09:02.5090055Z   test_wait_all_with_partial_exception (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17612
2022-05-18T04:09:02.5116564Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17613
2022-05-18T04:09:02.5142098Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17614
2022-05-18T04:09:02.5168166Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 17615
2022-05-18T04:09:03.2396915Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpje20adpz
2022-05-18T04:09:03.2398097Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpje20adpz/_remote_module_non_scriptable.py
2022-05-18T04:09:03.2595331Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn34fc55m
2022-05-18T04:09:03.2596194Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn34fc55m/_remote_module_non_scriptable.py
2022-05-18T04:09:03.2743271Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdthld9tb
2022-05-18T04:09:03.2744018Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdthld9tb/_remote_module_non_scriptable.py
2022-05-18T04:09:03.3276488Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq54g5fyh
2022-05-18T04:09:03.3277289Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq54g5fyh/_remote_module_non_scriptable.py
2022-05-18T04:09:03.5008016Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:09:03.5246780Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:09:03.5369117Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:09:03.5894566Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:09:03.8032387Z On WorkerInfo(id=1, name=worker1):
2022-05-18T04:09:03.8033201Z ValueError('Expected error')
2022-05-18T04:09:03.8033689Z Traceback (most recent call last):
2022-05-18T04:09:03.8034574Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function
2022-05-18T04:09:03.8035274Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-05-18T04:09:03.8036250Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 396, in raise_func
2022-05-18T04:09:03.8036911Z     raise ValueError(expected_err)
2022-05-18T04:09:03.8037329Z ValueError: Expected error
2022-05-18T04:09:03.8037591Z 
2022-05-18T04:09:03.8243361Z On WorkerInfo(id=0, name=worker0):
2022-05-18T04:09:03.8244345Z ValueError('Expected error')
2022-05-18T04:09:03.8244749Z Traceback (most recent call last):
2022-05-18T04:09:03.8245559Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function
2022-05-18T04:09:03.8246161Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-05-18T04:09:03.8246955Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 396, in raise_func
2022-05-18T04:09:03.8247438Z     raise ValueError(expected_err)
2022-05-18T04:09:03.8247807Z ValueError: Expected error
2022-05-18T04:09:03.8248090Z 
2022-05-18T04:09:03.8322874Z On WorkerInfo(id=3, name=worker3):
2022-05-18T04:09:03.8324516Z ValueError('Expected error')
2022-05-18T04:09:03.8325803Z Traceback (most recent call last):
2022-05-18T04:09:03.8326601Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function
2022-05-18T04:09:03.8327795Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-05-18T04:09:03.8329019Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 396, in raise_func
2022-05-18T04:09:03.8329577Z     raise ValueError(expected_err)
2022-05-18T04:09:03.8331723Z ValueError: Expected error
2022-05-18T04:09:03.8333751Z 
2022-05-18T04:09:03.8353338Z On WorkerInfo(id=2, name=worker2):
2022-05-18T04:09:03.8353842Z ValueError('Expected error')
2022-05-18T04:09:03.8354425Z Traceback (most recent call last):
2022-05-18T04:09:03.8355182Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function
2022-05-18T04:09:03.8355765Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-05-18T04:09:03.8356563Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 396, in raise_func
2022-05-18T04:09:03.8357139Z     raise ValueError(expected_err)
2022-05-18T04:09:03.8357549Z ValueError: Expected error
2022-05-18T04:09:03.8357771Z 
2022-05-18T04:09:04.1213532Z ok (1.960s)
2022-05-18T04:09:04.1213767Z 
2022-05-18T04:09:04.1214343Z ----------------------------------------------------------------------
2022-05-18T04:09:04.1214628Z Ran 1 test in 1.960s
2022-05-18T04:09:04.1214744Z 
2022-05-18T04:09:04.1214807Z OK
2022-05-18T04:09:04.1214900Z 
2022-05-18T04:09:04.1214995Z Generating XML reports...
2022-05-18T04:09:04.1248381Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040902.xml
2022-05-18T04:09:04.9870186Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpieinuxnc
2022-05-18T04:09:04.9871329Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpieinuxnc/_remote_module_non_scriptable.py
2022-05-18T04:09:05.2609983Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:09:05.2620023Z 
2022-05-18T04:09:05.2620204Z Running tests...
2022-05-18T04:09:05.2620547Z ----------------------------------------------------------------------
2022-05-18T04:09:05.6088141Z   test_wait_all_workers_dense (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17831
2022-05-18T04:09:05.6112026Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17832
2022-05-18T04:09:05.6135977Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17833
2022-05-18T04:09:05.6161515Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 17834
2022-05-18T04:09:06.2268897Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbrb18mu1
2022-05-18T04:09:06.2269672Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbrb18mu1/_remote_module_non_scriptable.py
2022-05-18T04:09:06.2367558Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxltrxr_p
2022-05-18T04:09:06.2369000Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxltrxr_p/_remote_module_non_scriptable.py
2022-05-18T04:09:06.2778425Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz7us9os4
2022-05-18T04:09:06.2780152Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz7us9os4/_remote_module_non_scriptable.py
2022-05-18T04:09:06.2861249Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgshvr4_n
2022-05-18T04:09:06.2862320Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgshvr4_n/_remote_module_non_scriptable.py
2022-05-18T04:09:06.4910769Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:09:06.4958374Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:09:06.5404808Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:09:06.5492716Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:09:06.5704875Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:09:06.5806365Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:09:06.5909223Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T04:09:06.5910132Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:09:06.5910732Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:09:06.5911480Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:09:06.5912317Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:09:06.5913169Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:09:07.1029471Z [W tensorpipe_agent.cpp:728] RPC agent for worker1 encountered error when reading incoming request from worker0: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2022-05-18T04:09:07.1030365Z [W tensorpipe_agent.cpp:728] RPC agent for worker3 encountered error when reading incoming request from worker0: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2022-05-18T04:09:07.1031229Z [W tensorpipe_agent.cpp:728] RPC agent for worker2 encountered error when reading incoming request from worker0: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2022-05-18T04:09:07.1033731Z [W tensorpipe_agent.cpp:728] RPC agent for worker0 encountered error when reading incoming request from worker1: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2022-05-18T04:09:07.1034983Z [W tensorpipe_agent.cpp:728] RPC agent for worker0 encountered error when reading incoming request from worker3: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2022-05-18T04:09:07.1036207Z [W tensorpipe_agent.cpp:728] RPC agent for worker0 encountered error when reading incoming request from worker2: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2022-05-18T04:09:07.3208427Z ok (2.058s)
2022-05-18T04:09:07.3208714Z 
2022-05-18T04:09:07.3209258Z ----------------------------------------------------------------------
2022-05-18T04:09:07.3209544Z Ran 1 test in 2.059s
2022-05-18T04:09:07.3209661Z 
2022-05-18T04:09:07.3209723Z OK
2022-05-18T04:09:07.3209816Z 
2022-05-18T04:09:07.3209912Z Generating XML reports...
2022-05-18T04:09:07.3245418Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040905.xml
2022-05-18T04:09:08.1956570Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8prpuwmp
2022-05-18T04:09:08.1957331Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8prpuwmp/_remote_module_non_scriptable.py
2022-05-18T04:09:08.4725430Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:09:08.4736210Z 
2022-05-18T04:09:08.4736568Z Running tests...
2022-05-18T04:09:08.4736969Z ----------------------------------------------------------------------
2022-05-18T04:09:08.8192672Z   test_wait_all_workers_timeout (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18062
2022-05-18T04:09:08.8215538Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18063
2022-05-18T04:09:08.8239648Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18064
2022-05-18T04:09:08.8265378Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 18065
2022-05-18T04:09:09.5135999Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph8wi_vny
2022-05-18T04:09:09.5136805Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph8wi_vny/_remote_module_non_scriptable.py
2022-05-18T04:09:09.5584483Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp26_kdoo
2022-05-18T04:09:09.5585261Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp26_kdoo/_remote_module_non_scriptable.py
2022-05-18T04:09:09.5799794Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd4yuf6_1
2022-05-18T04:09:09.5800748Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd4yuf6_1/_remote_module_non_scriptable.py
2022-05-18T04:09:09.6064329Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp39id0y2d
2022-05-18T04:09:09.6065077Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp39id0y2d/_remote_module_non_scriptable.py
2022-05-18T04:09:09.7760308Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:09:09.8194734Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:09:09.8425001Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:09:09.8678195Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:09:09.8877936Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:09:09.8939063Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:09:09.9042163Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:09:09.9043057Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:09:09.9043455Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T04:09:09.9043973Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:09:09.9044501Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:09:09.9081676Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:09:10.6334701Z [W tensorpipe_agent.cpp:942] RPC agent for worker0 encountered error when reading incoming response from worker1: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2022-05-18T04:09:10.6335953Z [W tensorpipe_agent.cpp:728] RPC agent for worker0 encountered error when reading incoming request from worker1: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2022-05-18T04:09:10.6336996Z [W tensorpipe_agent.cpp:728] RPC agent for worker1 encountered error when reading incoming request from worker0: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2022-05-18T04:09:10.6434820Z [W tensorpipe_agent.cpp:728] RPC agent for worker3 encountered error when reading incoming request from worker0: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2022-05-18T04:09:10.6435801Z [W tensorpipe_agent.cpp:728] RPC agent for worker0 encountered error when reading incoming request from worker3: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2022-05-18T04:09:10.6436822Z [W tensorpipe_agent.cpp:728] RPC agent for worker2 encountered error when reading incoming request from worker0: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2022-05-18T04:09:10.6439595Z [W tensorpipe_agent.cpp:728] RPC agent for worker0 encountered error when reading incoming request from worker2: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2022-05-18T04:09:11.1241240Z [W tensorpipe_agent.cpp:918] RPC agent for worker0 encountered error when sending outgoing request #4 to worker3: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2022-05-18T04:09:11.1242120Z [W tensorpipe_agent.cpp:627] RPC agent for worker1 won't send response to request #3 to worker0, as the agent is shutting down
2022-05-18T04:09:11.1242842Z [W tensorpipe_agent.cpp:681] RPC agent for worker0 encountered error when sending response to request #1 to worker1: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2022-05-18T04:09:11.1338541Z [W tensorpipe_agent.cpp:681] RPC agent for worker0 encountered error when sending response to request #1 to worker2: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2022-05-18T04:09:11.1344085Z [W tensorpipe_agent.cpp:681] RPC agent for worker0 encountered error when sending response to request #1 to worker3: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2022-05-18T04:09:11.6250432Z [W tensorpipe_agent.cpp:918] RPC agent for worker0 encountered error when sending outgoing request #5 to worker2: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2022-05-18T04:09:11.8331133Z ok (3.359s)
2022-05-18T04:09:11.8331359Z 
2022-05-18T04:09:11.8331883Z ----------------------------------------------------------------------
2022-05-18T04:09:11.8332255Z Ran 1 test in 3.359s
2022-05-18T04:09:11.8332370Z 
2022-05-18T04:09:11.8332430Z OK
2022-05-18T04:09:11.8332521Z 
2022-05-18T04:09:11.8332614Z Generating XML reports...
2022-05-18T04:09:11.8365575Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040908.xml
2022-05-18T04:09:12.7060862Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvsavycb0
2022-05-18T04:09:12.7061845Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvsavycb0/_remote_module_non_scriptable.py
2022-05-18T04:09:12.9739915Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:09:12.9750426Z 
2022-05-18T04:09:12.9750753Z Running tests...
2022-05-18T04:09:12.9751331Z ----------------------------------------------------------------------
2022-05-18T04:09:13.3226993Z   test_wait_all_workers_twice_dense (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18293
2022-05-18T04:09:13.3250364Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18294
2022-05-18T04:09:13.3274840Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18295
2022-05-18T04:09:13.3300695Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 18296
2022-05-18T04:09:14.0201406Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp61w2w3we
2022-05-18T04:09:14.0202197Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp61w2w3we/_remote_module_non_scriptable.py
2022-05-18T04:09:14.0334384Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6tznmp2j
2022-05-18T04:09:14.0335361Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6tznmp2j/_remote_module_non_scriptable.py
2022-05-18T04:09:14.0882105Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq1w3gzxp
2022-05-18T04:09:14.0883099Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq1w3gzxp/_remote_module_non_scriptable.py
2022-05-18T04:09:14.0973137Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp061gx53o
2022-05-18T04:09:14.0974100Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp061gx53o/_remote_module_non_scriptable.py
2022-05-18T04:09:14.2858602Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:09:14.2959960Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:09:14.3521593Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:09:14.3616584Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:09:14.3833455Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:09:14.3935349Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:09:14.3935983Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:09:14.3937100Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:09:14.3937836Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T04:09:14.3938776Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:09:14.3939414Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:09:14.3939938Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:09:14.8125692Z [W tensorpipe_agent.cpp:728] RPC agent for worker3 encountered error when reading incoming request from worker0: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2022-05-18T04:09:14.8126644Z [W tensorpipe_agent.cpp:728] RPC agent for worker0 encountered error when reading incoming request from worker1: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2022-05-18T04:09:14.8127490Z [W tensorpipe_agent.cpp:728] RPC agent for worker0 encountered error when reading incoming request from worker3: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2022-05-18T04:09:14.8128810Z [W tensorpipe_agent.cpp:728] RPC agent for worker0 encountered error when reading incoming request from worker2: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2022-05-18T04:09:14.8130261Z [W tensorpipe_agent.cpp:728] RPC agent for worker1 encountered error when reading incoming request from worker0: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2022-05-18T04:09:14.8136097Z [W tensorpipe_agent.cpp:728] RPC agent for worker2 encountered error when reading incoming request from worker0: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2022-05-18T04:09:15.0347046Z ok (2.059s)
2022-05-18T04:09:15.0347277Z 
2022-05-18T04:09:15.0347622Z ----------------------------------------------------------------------
2022-05-18T04:09:15.0347878Z Ran 1 test in 2.060s
2022-05-18T04:09:15.0347993Z 
2022-05-18T04:09:15.0348057Z OK
2022-05-18T04:09:15.0348143Z 
2022-05-18T04:09:15.0348240Z Generating XML reports...
2022-05-18T04:09:15.0385271Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040912.xml
2022-05-18T04:09:15.8795736Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnvybkb3c
2022-05-18T04:09:15.8796306Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnvybkb3c/_remote_module_non_scriptable.py
2022-05-18T04:09:16.1424392Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:09:16.1434366Z 
2022-05-18T04:09:16.1434467Z Running tests...
2022-05-18T04:09:16.1434951Z ----------------------------------------------------------------------
2022-05-18T04:09:16.4961627Z   test_without_world_size_existing_rank_can_communicate_with_new_rank (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18524
2022-05-18T04:09:16.4985943Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18525
2022-05-18T04:09:16.5010593Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18526
2022-05-18T04:09:16.5036813Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 18527
2022-05-18T04:09:17.1444742Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprc_qw597
2022-05-18T04:09:17.1445539Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprc_qw597/_remote_module_non_scriptable.py
2022-05-18T04:09:17.1484610Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeljfcs6e
2022-05-18T04:09:17.1485940Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeljfcs6e/_remote_module_non_scriptable.py
2022-05-18T04:09:17.1543618Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5cm_jsi3
2022-05-18T04:09:17.1544442Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5cm_jsi3/_remote_module_non_scriptable.py
2022-05-18T04:09:17.1558325Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsgy56i7m
2022-05-18T04:09:17.1559655Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsgy56i7m/_remote_module_non_scriptable.py
2022-05-18T04:09:17.4122679Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:09:17.4158042Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:09:17.4212057Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:09:17.4238999Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:09:17.4525331Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:09:17.4626581Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:09:17.4729455Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:09:17.4730042Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T04:09:17.4730937Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:09:17.4731779Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:09:17.4732821Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:09:17.4733616Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:09:17.9866568Z [W tensorpipe_agent.cpp:728] RPC agent for worker3 encountered error when reading incoming request from worker1: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2022-05-18T04:09:17.9867546Z [W tensorpipe_agent.cpp:728] RPC agent for worker0 encountered error when reading incoming request from worker1: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2022-05-18T04:09:17.9868184Z [W tensorpipe_agent.cpp:728] RPC agent for worker2 encountered error when reading incoming request from worker1: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2022-05-18T04:09:17.9965015Z [W tensorpipe_agent.cpp:728] RPC agent for worker2 encountered error when reading incoming request from worker3: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2022-05-18T04:09:17.9967323Z [W tensorpipe_agent.cpp:728] RPC agent for worker0 encountered error when reading incoming request from worker3: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2022-05-18T04:09:18.0059618Z [W tensorpipe_agent.cpp:728] RPC agent for worker0 encountered error when reading incoming request from worker2: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2022-05-18T04:09:18.3084879Z ok (2.165s)
2022-05-18T04:09:18.3085081Z 
2022-05-18T04:09:18.3085416Z ----------------------------------------------------------------------
2022-05-18T04:09:18.3085687Z Ran 1 test in 2.165s
2022-05-18T04:09:18.3085806Z 
2022-05-18T04:09:18.3085855Z OK
2022-05-18T04:09:18.3085949Z 
2022-05-18T04:09:18.3086046Z Generating XML reports...
2022-05-18T04:09:18.3120863Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040916.xml
2022-05-18T04:09:19.1731733Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyvod7wta
2022-05-18T04:09:19.1732234Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyvod7wta/_remote_module_non_scriptable.py
2022-05-18T04:09:19.4446684Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:09:19.4456535Z 
2022-05-18T04:09:19.4456682Z Running tests...
2022-05-18T04:09:19.4457066Z ----------------------------------------------------------------------
2022-05-18T04:09:19.7925713Z   test_without_world_size_existing_rank_can_communicate_with_new_rank_cuda (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18743
2022-05-18T04:09:19.7950346Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18744
2022-05-18T04:09:19.7974444Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18745
2022-05-18T04:09:19.7999990Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 18746
2022-05-18T04:09:20.4675075Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgw3pcrlk
2022-05-18T04:09:20.4675892Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgw3pcrlk/_remote_module_non_scriptable.py
2022-05-18T04:09:20.4916213Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6tkx25_1
2022-05-18T04:09:20.4917038Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6tkx25_1/_remote_module_non_scriptable.py
2022-05-18T04:09:20.5129457Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpoxr7k965
2022-05-18T04:09:20.5130284Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpoxr7k965/_remote_module_non_scriptable.py
2022-05-18T04:09:20.5518921Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcin101f5
2022-05-18T04:09:20.5519669Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcin101f5/_remote_module_non_scriptable.py
2022-05-18T04:09:20.7384977Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:09:20.7707819Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:09:20.7849309Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:09:20.8221947Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:09:21.0037920Z skip: Need at least 2 CUDA devices (1.558s)
2022-05-18T04:09:21.0038264Z 
2022-05-18T04:09:21.0039058Z ----------------------------------------------------------------------
2022-05-18T04:09:21.0039315Z Ran 1 test in 1.558s
2022-05-18T04:09:21.0039431Z 
2022-05-18T04:09:21.0039506Z OK (skipped=1)
2022-05-18T04:09:21.0039602Z 
2022-05-18T04:09:21.0039690Z Generating XML reports...
2022-05-18T04:09:21.0074199Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040919.xml
2022-05-18T04:09:21.8674428Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm7ap8e7q
2022-05-18T04:09:21.8675094Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm7ap8e7q/_remote_module_non_scriptable.py
2022-05-18T04:09:22.1365002Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:09:22.1375233Z 
2022-05-18T04:09:22.1375544Z Running tests...
2022-05-18T04:09:22.1375934Z ----------------------------------------------------------------------
2022-05-18T04:09:22.4892511Z   test_without_world_size_new_rank_can_communicated_with_existing_rank (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18798
2022-05-18T04:09:22.4916681Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18799
2022-05-18T04:09:22.4940170Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18800
2022-05-18T04:09:22.4965874Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 18801
2022-05-18T04:09:23.1228573Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5m6ajdk3
2022-05-18T04:09:23.1229768Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5m6ajdk3/_remote_module_non_scriptable.py
2022-05-18T04:09:23.1333212Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpatwsgcy3
2022-05-18T04:09:23.1333923Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpatwsgcy3/_remote_module_non_scriptable.py
2022-05-18T04:09:23.1361791Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjfqhv5z5
2022-05-18T04:09:23.1363338Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjfqhv5z5/_remote_module_non_scriptable.py
2022-05-18T04:09:23.1386688Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsgl_3n7v
2022-05-18T04:09:23.1388220Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsgl_3n7v/_remote_module_non_scriptable.py
2022-05-18T04:09:23.3845569Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:09:23.3956821Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:09:23.4019635Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:09:23.4022426Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:09:23.4237783Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:09:23.4238231Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:09:23.4339319Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:09:23.4340283Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T04:09:23.4340949Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:09:23.4341498Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:09:23.4342185Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:09:23.4342781Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:09:23.9523081Z [W tensorpipe_agent.cpp:728] RPC agent for worker2 encountered error when reading incoming request from worker0: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2022-05-18T04:09:23.9524094Z [W tensorpipe_agent.cpp:728] RPC agent for worker1 encountered error when reading incoming request from worker0: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2022-05-18T04:09:23.9525257Z [W tensorpipe_agent.cpp:728] RPC agent for worker3 encountered error when reading incoming request from worker0: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2022-05-18T04:09:23.9661850Z [W tensorpipe_agent.cpp:728] RPC agent for worker1 encountered error when reading incoming request from worker2: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2022-05-18T04:09:23.9663161Z [W tensorpipe_agent.cpp:728] RPC agent for worker3 encountered error when reading incoming request from worker2: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2022-05-18T04:09:23.9806023Z [W tensorpipe_agent.cpp:728] RPC agent for worker3 encountered error when reading incoming request from worker1: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2022-05-18T04:09:24.2013680Z ok (2.064s)
2022-05-18T04:09:24.2013967Z 
2022-05-18T04:09:24.2014474Z ----------------------------------------------------------------------
2022-05-18T04:09:24.2014738Z Ran 1 test in 2.064s
2022-05-18T04:09:24.2014856Z 
2022-05-18T04:09:24.2014905Z OK
2022-05-18T04:09:24.2015000Z 
2022-05-18T04:09:24.2015093Z Generating XML reports...
2022-05-18T04:09:24.2052819Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040922.xml
2022-05-18T04:09:25.0767537Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqlqjtl8q
2022-05-18T04:09:25.0768410Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqlqjtl8q/_remote_module_non_scriptable.py
2022-05-18T04:09:25.3473392Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:09:25.3483340Z 
2022-05-18T04:09:25.3483582Z Running tests...
2022-05-18T04:09:25.3484038Z ----------------------------------------------------------------------
2022-05-18T04:09:25.6947892Z   test_worker_id (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19017
2022-05-18T04:09:25.6973061Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19018
2022-05-18T04:09:25.6998054Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19019
2022-05-18T04:09:25.7024028Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 19020
2022-05-18T04:09:26.3108926Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmridarqm
2022-05-18T04:09:26.3109700Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmridarqm/_remote_module_non_scriptable.py
2022-05-18T04:09:26.3254518Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1wnpz40p
2022-05-18T04:09:26.3255777Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1wnpz40p/_remote_module_non_scriptable.py
2022-05-18T04:09:26.3387006Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgva0ww17
2022-05-18T04:09:26.3387837Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgva0ww17/_remote_module_non_scriptable.py
2022-05-18T04:09:26.3657407Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4es13ig7
2022-05-18T04:09:26.3658173Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4es13ig7/_remote_module_non_scriptable.py
2022-05-18T04:09:26.5762184Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:09:26.5902686Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:09:26.6016919Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:09:26.6306358Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:09:27.2067654Z ok (1.858s)
2022-05-18T04:09:27.2067926Z 
2022-05-18T04:09:27.2068409Z ----------------------------------------------------------------------
2022-05-18T04:09:27.2068669Z Ran 1 test in 1.858s
2022-05-18T04:09:27.2068786Z 
2022-05-18T04:09:27.2068851Z OK
2022-05-18T04:09:27.2068946Z 
2022-05-18T04:09:27.2069043Z Generating XML reports...
2022-05-18T04:09:27.2102819Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040925.xml
2022-05-18T04:09:28.0903234Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn2v5vmy7
2022-05-18T04:09:28.0903735Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn2v5vmy7/_remote_module_non_scriptable.py
2022-05-18T04:09:28.3602005Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:09:28.3612122Z 
2022-05-18T04:09:28.3612215Z Running tests...
2022-05-18T04:09:28.3612689Z ----------------------------------------------------------------------
2022-05-18T04:09:28.7111968Z   test_worker_info_pickle (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19236
2022-05-18T04:09:28.7137743Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19237
2022-05-18T04:09:28.7163208Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19238
2022-05-18T04:09:28.7189312Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 19239
2022-05-18T04:09:29.3119561Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgflqt153
2022-05-18T04:09:29.3120585Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgflqt153/_remote_module_non_scriptable.py
2022-05-18T04:09:29.3280732Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1fc2rjmo
2022-05-18T04:09:29.3281517Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1fc2rjmo/_remote_module_non_scriptable.py
2022-05-18T04:09:29.3632453Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf6sqk13j
2022-05-18T04:09:29.3633636Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf6sqk13j/_remote_module_non_scriptable.py
2022-05-18T04:09:29.3693518Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0kxvxanb
2022-05-18T04:09:29.3694650Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0kxvxanb/_remote_module_non_scriptable.py
2022-05-18T04:09:29.5759076Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:09:29.5883225Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:09:29.6258604Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:09:29.6295237Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:09:30.1230782Z ok (1.762s)
2022-05-18T04:09:30.1231016Z 
2022-05-18T04:09:30.1231560Z ----------------------------------------------------------------------
2022-05-18T04:09:30.1231890Z Ran 1 test in 1.762s
2022-05-18T04:09:30.1232007Z 
2022-05-18T04:09:30.1232055Z OK
2022-05-18T04:09:30.1232146Z 
2022-05-18T04:09:30.1232237Z Generating XML reports...
2022-05-18T04:09:30.1266285Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040928.xml
2022-05-18T04:09:30.9998969Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg82shjw0
2022-05-18T04:09:30.9999889Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg82shjw0/_remote_module_non_scriptable.py
2022-05-18T04:09:31.2709590Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:09:31.2719253Z 
2022-05-18T04:09:31.2719864Z Running tests...
2022-05-18T04:09:31.2720269Z ----------------------------------------------------------------------
2022-05-18T04:09:31.6177674Z   test_world_size_one (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19455
2022-05-18T04:09:31.6202949Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19456
2022-05-18T04:09:31.6227338Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19457
2022-05-18T04:09:31.6253417Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 19458
2022-05-18T04:09:32.3206266Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpexky467c
2022-05-18T04:09:32.3207106Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpexky467c/_remote_module_non_scriptable.py
2022-05-18T04:09:32.3580604Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpck4k83vo
2022-05-18T04:09:32.3581358Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpck4k83vo/_remote_module_non_scriptable.py
2022-05-18T04:09:32.3883838Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph7kb8zks
2022-05-18T04:09:32.3884519Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf_it7d9s
2022-05-18T04:09:32.3885157Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph7kb8zks/_remote_module_non_scriptable.py
2022-05-18T04:09:32.3885816Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf_it7d9s/_remote_module_non_scriptable.py
2022-05-18T04:09:32.5848016Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:09:32.6423247Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:09:32.6534832Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:09:32.6538283Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:09:32.9292578Z ok (1.657s)
2022-05-18T04:09:32.9292898Z 
2022-05-18T04:09:32.9293367Z ----------------------------------------------------------------------
2022-05-18T04:09:32.9293618Z Ran 1 test in 1.657s
2022-05-18T04:09:32.9293746Z 
2022-05-18T04:09:32.9293808Z OK
2022-05-18T04:09:32.9293886Z 
2022-05-18T04:09:32.9293980Z Generating XML reports...
2022-05-18T04:09:32.9330618Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040931.xml
2022-05-18T04:09:33.8083032Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx44002wr
2022-05-18T04:09:33.8083732Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx44002wr/_remote_module_non_scriptable.py
2022-05-18T04:09:34.0801404Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:09:34.0811354Z 
2022-05-18T04:09:34.0811445Z Running tests...
2022-05-18T04:09:34.0812423Z ----------------------------------------------------------------------
2022-05-18T04:09:34.4258552Z   test_wrong_types (__main__.TensorPipeRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19551
2022-05-18T04:09:34.4283909Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19552
2022-05-18T04:09:34.4309527Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19553
2022-05-18T04:09:34.4335812Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 19554
2022-05-18T04:09:35.0694156Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8zasena_
2022-05-18T04:09:35.0695147Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8zasena_/_remote_module_non_scriptable.py
2022-05-18T04:09:35.0820112Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa4807n6d
2022-05-18T04:09:35.0821148Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa4807n6d/_remote_module_non_scriptable.py
2022-05-18T04:09:35.0888280Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpumdrf28n
2022-05-18T04:09:35.0889462Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpumdrf28n/_remote_module_non_scriptable.py
2022-05-18T04:09:35.0943291Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe9xsh3sq
2022-05-18T04:09:35.0944577Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe9xsh3sq/_remote_module_non_scriptable.py
2022-05-18T04:09:35.3336211Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:09:35.3477093Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:09:35.3535025Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:09:35.3611205Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:09:35.5371624Z ok (1.456s)
2022-05-18T04:09:35.5371878Z 
2022-05-18T04:09:35.5372405Z ----------------------------------------------------------------------
2022-05-18T04:09:35.5372882Z Ran 1 test in 1.456s
2022-05-18T04:09:35.5373071Z 
2022-05-18T04:09:35.5373180Z OK
2022-05-18T04:09:35.5373351Z 
2022-05-18T04:09:35.5373516Z Generating XML reports...
2022-05-18T04:09:35.5410407Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040934.xml
2022-05-18T04:09:36.3954595Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeoywu0ds
2022-05-18T04:09:36.3955215Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeoywu0ds/_remote_module_non_scriptable.py
2022-05-18T04:09:36.6688504Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:09:36.6697881Z 
2022-05-18T04:09:36.6698019Z Running tests...
2022-05-18T04:09:36.6698628Z ----------------------------------------------------------------------
2022-05-18T04:09:37.0155066Z   test_backward_different_dtypes_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19606
2022-05-18T04:09:37.0179559Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19607
2022-05-18T04:09:37.0204736Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19608
2022-05-18T04:09:37.0231770Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 19609
2022-05-18T04:09:37.6919924Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjk0hxhln
2022-05-18T04:09:37.6920701Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjk0hxhln/_remote_module_non_scriptable.py
2022-05-18T04:09:37.7102802Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl9xsjv72
2022-05-18T04:09:37.7104139Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl9xsjv72/_remote_module_non_scriptable.py
2022-05-18T04:09:37.7105441Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd7kr0c5_
2022-05-18T04:09:37.7108638Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd7kr0c5_/_remote_module_non_scriptable.py
2022-05-18T04:09:37.7119604Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7t4lorqt
2022-05-18T04:09:37.7121463Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7t4lorqt/_remote_module_non_scriptable.py
2022-05-18T04:09:37.9556853Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:09:37.9716520Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:09:37.9721949Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:09:37.9722427Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:09:38.5273163Z ok (1.857s)
2022-05-18T04:09:38.5273364Z 
2022-05-18T04:09:38.5273675Z ----------------------------------------------------------------------
2022-05-18T04:09:38.5273936Z Ran 1 test in 1.857s
2022-05-18T04:09:38.5274052Z 
2022-05-18T04:09:38.5274115Z OK
2022-05-18T04:09:38.5274194Z 
2022-05-18T04:09:38.5274288Z Generating XML reports...
2022-05-18T04:09:38.5309231Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220518040936.xml
2022-05-18T04:09:39.4042798Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxerva7z3
2022-05-18T04:09:39.4043937Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxerva7z3/_remote_module_non_scriptable.py
2022-05-18T04:09:39.6756043Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:09:39.6765899Z 
2022-05-18T04:09:39.6766044Z Running tests...
2022-05-18T04:09:39.6766490Z ----------------------------------------------------------------------
2022-05-18T04:09:40.0240079Z   test_backward_multiple_round_trips_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19857
2022-05-18T04:09:40.0265681Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19858
2022-05-18T04:09:40.0290667Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19859
2022-05-18T04:09:40.0317255Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 19860
2022-05-18T04:09:40.7041766Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgh4k523j
2022-05-18T04:09:40.7042520Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2a6yzowg
2022-05-18T04:09:40.7043236Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgh4k523j/_remote_module_non_scriptable.py
2022-05-18T04:09:40.7045648Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2a6yzowg/_remote_module_non_scriptable.py
2022-05-18T04:09:40.7313815Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg822rcf2
2022-05-18T04:09:40.7315453Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg822rcf2/_remote_module_non_scriptable.py
2022-05-18T04:09:40.7433264Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr35bpoi7
2022-05-18T04:09:40.7434029Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr35bpoi7/_remote_module_non_scriptable.py
2022-05-18T04:09:40.9651805Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:09:40.9666385Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:09:40.9945870Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:09:41.0067213Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:09:41.7364675Z ok (2.059s)
2022-05-18T04:09:41.7364937Z 
2022-05-18T04:09:41.7365454Z ----------------------------------------------------------------------
2022-05-18T04:09:41.7365799Z Ran 1 test in 2.060s
2022-05-18T04:09:41.7365960Z 
2022-05-18T04:09:41.7366023Z OK
2022-05-18T04:09:41.7366118Z 
2022-05-18T04:09:41.7366214Z Generating XML reports...
2022-05-18T04:09:41.7400166Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220518040939.xml
2022-05-18T04:09:42.6209127Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbg03dksx
2022-05-18T04:09:42.6209969Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbg03dksx/_remote_module_non_scriptable.py
2022-05-18T04:09:42.8914856Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:09:42.8924568Z 
2022-05-18T04:09:42.8924728Z Running tests...
2022-05-18T04:09:42.8925148Z ----------------------------------------------------------------------
2022-05-18T04:09:43.2369881Z   test_backward_no_grad_on_tensor_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20156
2022-05-18T04:09:43.2394996Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20157
2022-05-18T04:09:43.2420056Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20158
2022-05-18T04:09:43.2447561Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 20159
2022-05-18T04:09:43.8680557Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpoq8p_dss
2022-05-18T04:09:43.8681579Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpoq8p_dss/_remote_module_non_scriptable.py
2022-05-18T04:09:43.8956310Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq82413nf
2022-05-18T04:09:43.8957113Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq82413nf/_remote_module_non_scriptable.py
2022-05-18T04:09:43.8958188Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpne4mzxoi
2022-05-18T04:09:43.8960598Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpne4mzxoi/_remote_module_non_scriptable.py
2022-05-18T04:09:43.9137274Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_1uvsh9m
2022-05-18T04:09:43.9138064Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_1uvsh9m/_remote_module_non_scriptable.py
2022-05-18T04:09:44.1313947Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:09:44.1578820Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:09:44.1599912Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:09:44.1795579Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:09:44.7490492Z ok (1.856s)
2022-05-18T04:09:44.7490632Z 
2022-05-18T04:09:44.7490986Z ----------------------------------------------------------------------
2022-05-18T04:09:44.7491271Z Ran 1 test in 1.856s
2022-05-18T04:09:44.7491390Z 
2022-05-18T04:09:44.7491453Z OK
2022-05-18T04:09:44.7491548Z 
2022-05-18T04:09:44.7491644Z Generating XML reports...
2022-05-18T04:09:44.7526037Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220518040942.xml
2022-05-18T04:09:45.6025257Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3cd8x40r
2022-05-18T04:09:45.6026080Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3cd8x40r/_remote_module_non_scriptable.py
2022-05-18T04:09:45.8721754Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:09:45.8731923Z 
2022-05-18T04:09:45.8732328Z Running tests...
2022-05-18T04:09:45.8732714Z ----------------------------------------------------------------------
2022-05-18T04:09:46.2186742Z   test_backward_rref_multi_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20407
2022-05-18T04:09:46.2211249Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20408
2022-05-18T04:09:46.2235006Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20409
2022-05-18T04:09:46.2261470Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 20410
2022-05-18T04:09:46.9338361Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpil9csvsa
2022-05-18T04:09:46.9340250Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpil9csvsa/_remote_module_non_scriptable.py
2022-05-18T04:09:46.9408916Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6qrxr_k_
2022-05-18T04:09:46.9410123Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6qrxr_k_/_remote_module_non_scriptable.py
2022-05-18T04:09:46.9748775Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb7ryv2jb
2022-05-18T04:09:46.9749488Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb7ryv2jb/_remote_module_non_scriptable.py
2022-05-18T04:09:46.9789592Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzixvndcx
2022-05-18T04:09:46.9791145Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzixvndcx/_remote_module_non_scriptable.py
2022-05-18T04:09:47.1933362Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:09:47.1996221Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:09:47.2349608Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:09:47.2396439Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:09:47.8305518Z ok (1.957s)
2022-05-18T04:09:47.8305730Z 
2022-05-18T04:09:47.8306246Z ----------------------------------------------------------------------
2022-05-18T04:09:47.8306687Z Ran 1 test in 1.957s
2022-05-18T04:09:47.8306895Z 
2022-05-18T04:09:47.8307013Z OK
2022-05-18T04:09:47.8307210Z 
2022-05-18T04:09:47.8307360Z Generating XML reports...
2022-05-18T04:09:47.8341015Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220518040945.xml
2022-05-18T04:09:48.6093220Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptk4pqu1z
2022-05-18T04:09:48.6094164Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptk4pqu1z/_remote_module_non_scriptable.py
2022-05-18T04:09:48.8633008Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:09:48.8642578Z 
2022-05-18T04:09:48.8642684Z Running tests...
2022-05-18T04:09:48.8643074Z ----------------------------------------------------------------------
2022-05-18T04:09:49.1817894Z   test_backward_rref_nested_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20667
2022-05-18T04:09:49.1840717Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20668
2022-05-18T04:09:49.1864687Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20669
2022-05-18T04:09:49.1888829Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 20670
2022-05-18T04:09:49.7960079Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo4_dwzc0
2022-05-18T04:09:49.7960851Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo4_dwzc0/_remote_module_non_scriptable.py
2022-05-18T04:09:49.8071896Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkqxv1c4l
2022-05-18T04:09:49.8072958Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkqxv1c4l/_remote_module_non_scriptable.py
2022-05-18T04:09:49.8250540Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp59qym7tc
2022-05-18T04:09:49.8251697Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp59qym7tc/_remote_module_non_scriptable.py
2022-05-18T04:09:49.8269542Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptl_pvv6a
2022-05-18T04:09:49.8271928Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptl_pvv6a/_remote_module_non_scriptable.py
2022-05-18T04:09:50.0449167Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:09:50.0540069Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:09:50.0728448Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:09:50.0754925Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:09:50.6931314Z ok (1.829s)
2022-05-18T04:09:50.6931593Z 
2022-05-18T04:09:50.6932138Z ----------------------------------------------------------------------
2022-05-18T04:09:50.6932558Z Ran 1 test in 1.829s
2022-05-18T04:09:50.6932687Z 
2022-05-18T04:09:50.6932749Z OK
2022-05-18T04:09:50.6932843Z 
2022-05-18T04:09:50.6932937Z Generating XML reports...
2022-05-18T04:09:50.6966629Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220518040948.xml
2022-05-18T04:09:51.4629947Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2akimvyl
2022-05-18T04:09:51.4630633Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2akimvyl/_remote_module_non_scriptable.py
2022-05-18T04:09:51.7158694Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:09:51.7168963Z 
2022-05-18T04:09:51.7169392Z Running tests...
2022-05-18T04:09:51.7169800Z ----------------------------------------------------------------------
2022-05-18T04:09:52.0312174Z   test_backward_rref_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20942
2022-05-18T04:09:52.0334980Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20943
2022-05-18T04:09:52.0358325Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20944
2022-05-18T04:09:52.0382633Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 20945
2022-05-18T04:09:52.7097204Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9q1j9uxv
2022-05-18T04:09:52.7098490Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9q1j9uxv/_remote_module_non_scriptable.py
2022-05-18T04:09:52.7203528Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpaqlr0enk
2022-05-18T04:09:52.7204799Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpaqlr0enk/_remote_module_non_scriptable.py
2022-05-18T04:09:52.7831994Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz1peg705
2022-05-18T04:09:52.7833035Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz1peg705/_remote_module_non_scriptable.py
2022-05-18T04:09:52.8654341Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy2uhprda
2022-05-18T04:09:52.8655149Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy2uhprda/_remote_module_non_scriptable.py
2022-05-18T04:09:52.9565969Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:09:52.9688799Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:09:53.0567579Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:09:53.1127747Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:09:53.6425718Z ok (1.925s)
2022-05-18T04:09:53.6425959Z 
2022-05-18T04:09:53.6426711Z ----------------------------------------------------------------------
2022-05-18T04:09:53.6427449Z Ran 1 test in 1.926s
2022-05-18T04:09:53.6427567Z 
2022-05-18T04:09:53.6427630Z OK
2022-05-18T04:09:53.6427722Z 
2022-05-18T04:09:53.6427815Z Generating XML reports...
2022-05-18T04:09:53.6461341Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220518040951.xml
2022-05-18T04:09:54.4113388Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0fc_vm9k
2022-05-18T04:09:54.4114283Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0fc_vm9k/_remote_module_non_scriptable.py
2022-05-18T04:09:54.6645585Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:09:54.6656314Z 
2022-05-18T04:09:54.6656533Z Running tests...
2022-05-18T04:09:54.6656919Z ----------------------------------------------------------------------
2022-05-18T04:09:54.9786380Z   test_backward_simple_python_udf_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21217
2022-05-18T04:09:54.9809479Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21218
2022-05-18T04:09:54.9832656Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21219
2022-05-18T04:09:54.9856808Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 21220
2022-05-18T04:09:55.6045115Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq78r26sv
2022-05-18T04:09:55.6045873Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq78r26sv/_remote_module_non_scriptable.py
2022-05-18T04:09:55.6152507Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvx46y_os
2022-05-18T04:09:55.6153499Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvx46y_os/_remote_module_non_scriptable.py
2022-05-18T04:09:55.6218082Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyn3xvnp4
2022-05-18T04:09:55.6219417Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyn3xvnp4/_remote_module_non_scriptable.py
2022-05-18T04:09:55.6220226Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbqy2xn19
2022-05-18T04:09:55.6224165Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbqy2xn19/_remote_module_non_scriptable.py
2022-05-18T04:09:55.8516780Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:09:55.8620653Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:09:55.8700313Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:09:55.8703317Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:09:56.3897077Z ok (1.724s)
2022-05-18T04:09:56.3897367Z 
2022-05-18T04:09:56.3897885Z ----------------------------------------------------------------------
2022-05-18T04:09:56.3898209Z Ran 1 test in 1.724s
2022-05-18T04:09:56.3898312Z 
2022-05-18T04:09:56.3898375Z OK
2022-05-18T04:09:56.3898466Z 
2022-05-18T04:09:56.3898564Z Generating XML reports...
2022-05-18T04:09:56.3931830Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220518040954.xml
2022-05-18T04:09:57.1585035Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjgnfnmt8
2022-05-18T04:09:57.1586080Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjgnfnmt8/_remote_module_non_scriptable.py
2022-05-18T04:09:57.4127352Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:09:57.4136870Z 
2022-05-18T04:09:57.4136977Z Running tests...
2022-05-18T04:09:57.4137531Z ----------------------------------------------------------------------
2022-05-18T04:09:57.7305540Z   test_backward_simple_script_call_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21468
2022-05-18T04:09:57.7328256Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21469
2022-05-18T04:09:57.7351465Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21470
2022-05-18T04:09:57.7375175Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 21471
2022-05-18T04:09:58.3345723Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc3s8kol9
2022-05-18T04:09:58.3346558Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc3s8kol9/_remote_module_non_scriptable.py
2022-05-18T04:09:58.3419950Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppy550ugu
2022-05-18T04:09:58.3420934Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppy550ugu/_remote_module_non_scriptable.py
2022-05-18T04:09:58.3683703Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf_osmr0e
2022-05-18T04:09:58.3684584Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf_osmr0e/_remote_module_non_scriptable.py
2022-05-18T04:09:58.3750311Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwgw7ggy_
2022-05-18T04:09:58.3751619Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwgw7ggy_/_remote_module_non_scriptable.py
2022-05-18T04:09:58.5826275Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:09:58.5876068Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:09:58.6161171Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:09:58.6217675Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:09:59.3419337Z ok (1.928s)
2022-05-18T04:09:59.3419702Z 
2022-05-18T04:09:59.3420068Z ----------------------------------------------------------------------
2022-05-18T04:09:59.3420307Z Ran 1 test in 1.928s
2022-05-18T04:09:59.3420427Z 
2022-05-18T04:09:59.3420487Z OK
2022-05-18T04:09:59.3420588Z 
2022-05-18T04:09:59.3420680Z Generating XML reports...
2022-05-18T04:09:59.3454018Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220518040957.xml
2022-05-18T04:10:00.1071619Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpiepi81d5
2022-05-18T04:10:00.1072418Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpiepi81d5/_remote_module_non_scriptable.py
2022-05-18T04:10:00.3604081Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:10:00.3613511Z 
2022-05-18T04:10:00.3613638Z Running tests...
2022-05-18T04:10:00.3614128Z ----------------------------------------------------------------------
2022-05-18T04:10:00.6750437Z   test_backward_simple_self_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21719
2022-05-18T04:10:00.6772802Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21720
2022-05-18T04:10:00.6795715Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21721
2022-05-18T04:10:00.6820676Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 21722
2022-05-18T04:10:01.2666797Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvz4ox5__
2022-05-18T04:10:01.2667718Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvz4ox5__/_remote_module_non_scriptable.py
2022-05-18T04:10:01.3061278Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr1zts41h
2022-05-18T04:10:01.3062159Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr1zts41h/_remote_module_non_scriptable.py
2022-05-18T04:10:01.3177506Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_k7jj3to
2022-05-18T04:10:01.3179242Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_k7jj3to/_remote_module_non_scriptable.py
2022-05-18T04:10:01.3298516Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpikl1zhod
2022-05-18T04:10:01.3299276Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpikl1zhod/_remote_module_non_scriptable.py
2022-05-18T04:10:01.5183525Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:10:01.5563754Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:10:01.5675601Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:10:01.5793423Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:10:02.1863811Z ok (1.825s)
2022-05-18T04:10:02.1864034Z 
2022-05-18T04:10:02.1864409Z ----------------------------------------------------------------------
2022-05-18T04:10:02.1864737Z Ran 1 test in 1.825s
2022-05-18T04:10:02.1864881Z 
2022-05-18T04:10:02.1864930Z OK
2022-05-18T04:10:02.1865026Z 
2022-05-18T04:10:02.1865123Z Generating XML reports...
2022-05-18T04:10:02.1899006Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220518041000.xml
2022-05-18T04:10:02.9797896Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2u5lrdw3
2022-05-18T04:10:02.9798654Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2u5lrdw3/_remote_module_non_scriptable.py
2022-05-18T04:10:03.2319785Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:10:03.2329600Z 
2022-05-18T04:10:03.2329693Z Running tests...
2022-05-18T04:10:03.2330738Z ----------------------------------------------------------------------
2022-05-18T04:10:03.5506574Z   test_backward_simple_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21970
2022-05-18T04:10:03.5529593Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21971
2022-05-18T04:10:03.5551169Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21972
2022-05-18T04:10:03.5575850Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 21973
2022-05-18T04:10:04.2695992Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo_znakvr
2022-05-18T04:10:04.2697227Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo_znakvr/_remote_module_non_scriptable.py
2022-05-18T04:10:04.2913752Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplzm60moi
2022-05-18T04:10:04.2916229Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplzm60moi/_remote_module_non_scriptable.py
2022-05-18T04:10:04.2974498Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf3a37gs4
2022-05-18T04:10:04.2976766Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf3a37gs4/_remote_module_non_scriptable.py
2022-05-18T04:10:04.3205271Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdpv5k7tb
2022-05-18T04:10:04.3206916Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdpv5k7tb/_remote_module_non_scriptable.py
2022-05-18T04:10:04.5177378Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:10:04.5446553Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:10:04.5501122Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:10:04.5678736Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:10:05.0616227Z ok (1.828s)
2022-05-18T04:10:05.0616396Z 
2022-05-18T04:10:05.0616783Z ----------------------------------------------------------------------
2022-05-18T04:10:05.0617276Z Ran 1 test in 1.829s
2022-05-18T04:10:05.0617396Z 
2022-05-18T04:10:05.0617449Z OK
2022-05-18T04:10:05.0617542Z 
2022-05-18T04:10:05.0617638Z Generating XML reports...
2022-05-18T04:10:05.0653576Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220518041003.xml
2022-05-18T04:10:05.8306864Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5tdv8v9k
2022-05-18T04:10:05.8307453Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5tdv8v9k/_remote_module_non_scriptable.py
2022-05-18T04:10:06.0818696Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:10:06.0828715Z 
2022-05-18T04:10:06.0829162Z Running tests...
2022-05-18T04:10:06.0829740Z ----------------------------------------------------------------------
2022-05-18T04:10:06.3942838Z   test_backwards_nested_python_udf_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22221
2022-05-18T04:10:06.3965210Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22222
2022-05-18T04:10:06.3988451Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22223
2022-05-18T04:10:06.4012334Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 22224
2022-05-18T04:10:07.0840756Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp022gsx1j
2022-05-18T04:10:07.0841535Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp022gsx1j/_remote_module_non_scriptable.py
2022-05-18T04:10:07.0940984Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplrrdlyix
2022-05-18T04:10:07.0941739Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx0wgdcrg
2022-05-18T04:10:07.0942445Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplrrdlyix/_remote_module_non_scriptable.py
2022-05-18T04:10:07.0943329Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx0wgdcrg/_remote_module_non_scriptable.py
2022-05-18T04:10:07.0979321Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk37zkq6n
2022-05-18T04:10:07.0980806Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk37zkq6n/_remote_module_non_scriptable.py
2022-05-18T04:10:07.3336168Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:10:07.3433492Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:10:07.3456554Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:10:07.3471955Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:10:07.9054419Z ok (1.822s)
2022-05-18T04:10:07.9054647Z 
2022-05-18T04:10:07.9055007Z ----------------------------------------------------------------------
2022-05-18T04:10:07.9055264Z Ran 1 test in 1.822s
2022-05-18T04:10:07.9055383Z 
2022-05-18T04:10:07.9055457Z OK
2022-05-18T04:10:07.9055585Z 
2022-05-18T04:10:07.9055679Z Generating XML reports...
2022-05-18T04:10:07.9088850Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220518041006.xml
2022-05-18T04:10:08.6750949Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeafjosrx
2022-05-18T04:10:08.6751692Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeafjosrx/_remote_module_non_scriptable.py
2022-05-18T04:10:08.9253102Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:10:08.9263079Z 
2022-05-18T04:10:08.9263205Z Running tests...
2022-05-18T04:10:08.9263810Z ----------------------------------------------------------------------
2022-05-18T04:10:09.2392877Z   test_context_cleanup_nested_rpc_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22472
2022-05-18T04:10:09.2414441Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22473
2022-05-18T04:10:09.2437879Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22474
2022-05-18T04:10:09.2462343Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 22475
2022-05-18T04:10:09.8857433Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp71_ch1xb
2022-05-18T04:10:09.8858420Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp71_ch1xb/_remote_module_non_scriptable.py
2022-05-18T04:10:09.9023773Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa15iz61w
2022-05-18T04:10:09.9024520Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa15iz61w/_remote_module_non_scriptable.py
2022-05-18T04:10:09.9291168Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxuulncsf
2022-05-18T04:10:09.9292159Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxuulncsf/_remote_module_non_scriptable.py
2022-05-18T04:10:09.9557271Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8jjf8llx
2022-05-18T04:10:09.9558274Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8jjf8llx/_remote_module_non_scriptable.py
2022-05-18T04:10:10.1338184Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:10:10.1525242Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:10:10.1767900Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:10:10.2062573Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:10:10.4694200Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:10:10.4788914Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:10:10.4888503Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:10:10.4890211Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:10:10.4890930Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T04:10:10.4892870Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:10:10.4893796Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:10:10.4897511Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:10:10.8506458Z ok (1.924s)
2022-05-18T04:10:10.8506759Z 
2022-05-18T04:10:10.8507302Z ----------------------------------------------------------------------
2022-05-18T04:10:10.8507599Z Ran 1 test in 1.924s
2022-05-18T04:10:10.8507719Z 
2022-05-18T04:10:10.8507780Z OK
2022-05-18T04:10:10.8507861Z 
2022-05-18T04:10:10.8507956Z Generating XML reports...
2022-05-18T04:10:10.8540882Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220518041008.xml
2022-05-18T04:10:11.6171287Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp87_wk9gd
2022-05-18T04:10:11.6172043Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp87_wk9gd/_remote_module_non_scriptable.py
2022-05-18T04:10:11.8728051Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:10:11.8737351Z 
2022-05-18T04:10:11.8737464Z Running tests...
2022-05-18T04:10:11.8738353Z ----------------------------------------------------------------------
2022-05-18T04:10:12.1919535Z   test_context_cleanup_tensor_no_grad_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22715
2022-05-18T04:10:12.1942457Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22716
2022-05-18T04:10:12.1965558Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22717
2022-05-18T04:10:12.1989805Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 22718
2022-05-18T04:10:12.7785235Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4vfbi49i
2022-05-18T04:10:12.7786954Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4vfbi49i/_remote_module_non_scriptable.py
2022-05-18T04:10:12.7793976Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgqn0cz6g
2022-05-18T04:10:12.7796165Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgqn0cz6g/_remote_module_non_scriptable.py
2022-05-18T04:10:12.8173665Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpur5desqc
2022-05-18T04:10:12.8175162Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpur5desqc/_remote_module_non_scriptable.py
2022-05-18T04:10:12.8345829Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfbnj57po
2022-05-18T04:10:12.8347080Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfbnj57po/_remote_module_non_scriptable.py
2022-05-18T04:10:13.0277081Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:10:13.0286930Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:10:13.0669661Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:10:13.0827236Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:10:13.3776089Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:10:13.3833452Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:10:13.3834476Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:10:13.3835513Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:10:13.3836349Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:10:13.3836758Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T04:10:13.3837261Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:10:13.3876980Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:10:13.8035398Z ok (1.929s)
2022-05-18T04:10:13.8035545Z 
2022-05-18T04:10:13.8035948Z ----------------------------------------------------------------------
2022-05-18T04:10:13.8036221Z Ran 1 test in 1.930s
2022-05-18T04:10:13.8036337Z 
2022-05-18T04:10:13.8036400Z OK
2022-05-18T04:10:13.8036493Z 
2022-05-18T04:10:13.8036576Z Generating XML reports...
2022-05-18T04:10:13.8070188Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220518041011.xml
2022-05-18T04:10:14.5805697Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptq6jofcm
2022-05-18T04:10:14.5806653Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptq6jofcm/_remote_module_non_scriptable.py
2022-05-18T04:10:14.8347847Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:10:14.8358015Z 
2022-05-18T04:10:14.8358639Z Running tests...
2022-05-18T04:10:14.8359103Z ----------------------------------------------------------------------
2022-05-18T04:10:15.1527242Z   test_context_cleanup_tensor_with_grad_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22958
2022-05-18T04:10:15.1550416Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22959
2022-05-18T04:10:15.1573579Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22960
2022-05-18T04:10:15.1598588Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 22961
2022-05-18T04:10:15.8347395Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp54opp05n
2022-05-18T04:10:15.8348280Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp54opp05n/_remote_module_non_scriptable.py
2022-05-18T04:10:15.8456771Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc50k0w62
2022-05-18T04:10:15.8457735Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc50k0w62/_remote_module_non_scriptable.py
2022-05-18T04:10:15.9075525Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2v6yp5o3
2022-05-18T04:10:15.9076295Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2v6yp5o3/_remote_module_non_scriptable.py
2022-05-18T04:10:15.9093879Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp69ibp2ty
2022-05-18T04:10:15.9095235Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp69ibp2ty/_remote_module_non_scriptable.py
2022-05-18T04:10:16.0918009Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:10:16.1027024Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:10:16.1666172Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:10:16.1709206Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:10:16.4418999Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:10:16.4522655Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:10:16.4624061Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:10:16.4625700Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T04:10:16.4628268Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:10:16.4629756Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:10:16.4631138Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:10:16.4633507Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:10:16.8646141Z ok (2.028s)
2022-05-18T04:10:16.8646427Z 
2022-05-18T04:10:16.8646898Z ----------------------------------------------------------------------
2022-05-18T04:10:16.8647212Z Ran 1 test in 2.029s
2022-05-18T04:10:16.8647330Z 
2022-05-18T04:10:16.8647393Z OK
2022-05-18T04:10:16.8647486Z 
2022-05-18T04:10:16.8647584Z Generating XML reports...
2022-05-18T04:10:16.8685534Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220518041014.xml
2022-05-18T04:10:17.6962881Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbc_x0hej
2022-05-18T04:10:17.6963714Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbc_x0hej/_remote_module_non_scriptable.py
2022-05-18T04:10:17.9566363Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:10:17.9575763Z 
2022-05-18T04:10:17.9575899Z Running tests...
2022-05-18T04:10:17.9576492Z ----------------------------------------------------------------------
2022-05-18T04:10:18.3015960Z   test_embedding_bag_with_no_grad_tensors (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23201
2022-05-18T04:10:18.3041441Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23202
2022-05-18T04:10:18.3066293Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23203
2022-05-18T04:10:18.3091054Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 23204
2022-05-18T04:10:18.9407399Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzmyrm4rn
2022-05-18T04:10:18.9408695Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzmyrm4rn/_remote_module_non_scriptable.py
2022-05-18T04:10:18.9456690Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp_0exqrb
2022-05-18T04:10:18.9457829Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp_0exqrb/_remote_module_non_scriptable.py
2022-05-18T04:10:18.9677021Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa6g66wrq
2022-05-18T04:10:18.9678093Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa6g66wrq/_remote_module_non_scriptable.py
2022-05-18T04:10:18.9725489Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp864e6q81
2022-05-18T04:10:18.9726382Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp864e6q81/_remote_module_non_scriptable.py
2022-05-18T04:10:19.2044402Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:10:19.2048587Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:10:19.2298236Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:10:19.2345050Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:10:19.8135814Z ok (1.856s)
2022-05-18T04:10:19.8136077Z 
2022-05-18T04:10:19.8136553Z ----------------------------------------------------------------------
2022-05-18T04:10:19.8136810Z Ran 1 test in 1.856s
2022-05-18T04:10:19.8136941Z 
2022-05-18T04:10:19.8137003Z OK
2022-05-18T04:10:19.8137095Z 
2022-05-18T04:10:19.8137190Z Generating XML reports...
2022-05-18T04:10:19.8171798Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220518041017.xml
2022-05-18T04:10:20.6581963Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8ku0t2k6
2022-05-18T04:10:20.6582438Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8ku0t2k6/_remote_module_non_scriptable.py
2022-05-18T04:10:20.9201985Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:10:20.9211547Z 
2022-05-18T04:10:20.9211878Z Running tests...
2022-05-18T04:10:20.9212300Z ----------------------------------------------------------------------
2022-05-18T04:10:21.2469632Z   test_graph_for_builtin_call_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23464
2022-05-18T04:10:21.2492457Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23465
2022-05-18T04:10:21.2515699Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23466
2022-05-18T04:10:21.2539964Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 23467
2022-05-18T04:10:21.9239141Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr3l6z44d
2022-05-18T04:10:21.9240012Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr3l6z44d/_remote_module_non_scriptable.py
2022-05-18T04:10:21.9247708Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbr_vj53i
2022-05-18T04:10:21.9249671Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbr_vj53i/_remote_module_non_scriptable.py
2022-05-18T04:10:21.9390786Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6wn7dy8j
2022-05-18T04:10:21.9404111Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6wn7dy8j/_remote_module_non_scriptable.py
2022-05-18T04:10:21.9828598Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph0bjlz0_
2022-05-18T04:10:21.9829445Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph0bjlz0_/_remote_module_non_scriptable.py
2022-05-18T04:10:22.1782235Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:10:22.1819033Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:10:22.1946061Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:10:22.2388866Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:10:22.5036581Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:10:22.5136520Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:10:22.5238479Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:10:22.5241737Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T04:10:22.5245454Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:10:22.5247798Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:10:22.5249430Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:10:22.5254896Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:10:23.0588706Z ok (2.137s)
2022-05-18T04:10:23.0588978Z 
2022-05-18T04:10:23.0589500Z ----------------------------------------------------------------------
2022-05-18T04:10:23.0589823Z Ran 1 test in 2.138s
2022-05-18T04:10:23.0589963Z 
2022-05-18T04:10:23.0590023Z OK
2022-05-18T04:10:23.0590115Z 
2022-05-18T04:10:23.0590212Z Generating XML reports...
2022-05-18T04:10:23.0624280Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220518041020.xml
2022-05-18T04:10:23.8931389Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphxormzbd
2022-05-18T04:10:23.8932398Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphxormzbd/_remote_module_non_scriptable.py
2022-05-18T04:10:24.1511833Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:10:24.1521769Z 
2022-05-18T04:10:24.1521996Z Running tests...
2022-05-18T04:10:24.1522407Z ----------------------------------------------------------------------
2022-05-18T04:10:24.4829386Z   test_graph_for_builtin_remote_call_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23707
2022-05-18T04:10:24.4853609Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23708
2022-05-18T04:10:24.4878475Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23709
2022-05-18T04:10:24.4903364Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 23710
2022-05-18T04:10:25.1231592Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdlx_4hat
2022-05-18T04:10:25.1232582Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdlx_4hat/_remote_module_non_scriptable.py
2022-05-18T04:10:25.1536799Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf29x2sim
2022-05-18T04:10:25.1537520Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf29x2sim/_remote_module_non_scriptable.py
2022-05-18T04:10:25.1922136Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5o9zy0qh
2022-05-18T04:10:25.1923533Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5o9zy0qh/_remote_module_non_scriptable.py
2022-05-18T04:10:25.1926145Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjzu9sqdt
2022-05-18T04:10:25.1927892Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjzu9sqdt/_remote_module_non_scriptable.py
2022-05-18T04:10:25.3811878Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:10:25.4105588Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:10:25.4472460Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:10:25.4473160Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:10:25.6936908Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:10:25.6937759Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:10:25.7038379Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T04:10:25.7040733Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:10:25.7043357Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:10:25.7045765Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:10:25.7049206Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:10:25.7050557Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:10:26.1950356Z ok (2.043s)
2022-05-18T04:10:26.1950591Z 
2022-05-18T04:10:26.1951230Z ----------------------------------------------------------------------
2022-05-18T04:10:26.1951657Z Ran 1 test in 2.043s
2022-05-18T04:10:26.1951859Z 
2022-05-18T04:10:26.1951955Z OK
2022-05-18T04:10:26.1952112Z 
2022-05-18T04:10:26.1952261Z Generating XML reports...
2022-05-18T04:10:26.1987581Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220518041024.xml
2022-05-18T04:10:27.0543113Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy9mqzyuo
2022-05-18T04:10:27.0544131Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy9mqzyuo/_remote_module_non_scriptable.py
2022-05-18T04:10:27.3221257Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:10:27.3231452Z 
2022-05-18T04:10:27.3231578Z Running tests...
2022-05-18T04:10:27.3232010Z ----------------------------------------------------------------------
2022-05-18T04:10:27.6678045Z   test_graph_for_py_nested_call_itself_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23950
2022-05-18T04:10:27.6702765Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23951
2022-05-18T04:10:27.6727762Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23952
2022-05-18T04:10:27.6753821Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 23953
2022-05-18T04:10:28.3286933Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfg6rjojp
2022-05-18T04:10:28.3287727Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfg6rjojp/_remote_module_non_scriptable.py
2022-05-18T04:10:28.3565998Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsqdq74pj
2022-05-18T04:10:28.3567232Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsqdq74pj/_remote_module_non_scriptable.py
2022-05-18T04:10:28.3761543Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp35jisbxi
2022-05-18T04:10:28.3762341Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp35jisbxi/_remote_module_non_scriptable.py
2022-05-18T04:10:28.3807525Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphmgiagvo
2022-05-18T04:10:28.3808426Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphmgiagvo/_remote_module_non_scriptable.py
2022-05-18T04:10:28.5883417Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:10:28.6164012Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:10:28.6346566Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:10:28.6391009Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:10:28.9003697Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:10:28.9004593Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:10:28.9105524Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:10:28.9115334Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:10:28.9116245Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T04:10:28.9117350Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:10:28.9118512Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:10:28.9119633Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:10:29.3800144Z ok (2.056s)
2022-05-18T04:10:29.3800401Z 
2022-05-18T04:10:29.3800856Z ----------------------------------------------------------------------
2022-05-18T04:10:29.3801094Z Ran 1 test in 2.057s
2022-05-18T04:10:29.3801210Z 
2022-05-18T04:10:29.3801270Z OK
2022-05-18T04:10:29.3801361Z 
2022-05-18T04:10:29.3801455Z Generating XML reports...
2022-05-18T04:10:29.3836006Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220518041027.xml
2022-05-18T04:10:30.2515860Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbkofzzob
2022-05-18T04:10:30.2516679Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbkofzzob/_remote_module_non_scriptable.py
2022-05-18T04:10:30.5181218Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:10:30.5191438Z 
2022-05-18T04:10:30.5191617Z Running tests...
2022-05-18T04:10:30.5191984Z ----------------------------------------------------------------------
2022-05-18T04:10:30.8646098Z   test_graph_for_py_nested_call_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24193
2022-05-18T04:10:30.8671591Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24194
2022-05-18T04:10:30.8696238Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24195
2022-05-18T04:10:30.8722247Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 24196
2022-05-18T04:10:31.4771287Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp09dtke6f
2022-05-18T04:10:31.4772280Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp09dtke6f/_remote_module_non_scriptable.py
2022-05-18T04:10:31.4959520Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd07rhvu1
2022-05-18T04:10:31.4960318Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd07rhvu1/_remote_module_non_scriptable.py
2022-05-18T04:10:31.5275080Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2n5z48_4
2022-05-18T04:10:31.5276451Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2n5z48_4/_remote_module_non_scriptable.py
2022-05-18T04:10:31.5522150Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8uvqyd3h
2022-05-18T04:10:31.5522891Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8uvqyd3h/_remote_module_non_scriptable.py
2022-05-18T04:10:31.7414286Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:10:31.7623435Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:10:31.7907586Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:10:31.8139585Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:10:32.0772579Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:10:32.0875534Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:10:32.0876283Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:10:32.0877356Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:10:32.0877831Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T04:10:32.0878505Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:10:32.0879734Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:10:32.0979538Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:10:32.4767224Z ok (1.957s)
2022-05-18T04:10:32.4767455Z 
2022-05-18T04:10:32.4767993Z ----------------------------------------------------------------------
2022-05-18T04:10:32.4768245Z Ran 1 test in 1.957s
2022-05-18T04:10:32.4768359Z 
2022-05-18T04:10:32.4768420Z OK
2022-05-18T04:10:32.4768512Z 
2022-05-18T04:10:32.4768594Z Generating XML reports...
2022-05-18T04:10:32.4803815Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220518041030.xml
2022-05-18T04:10:33.3371431Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpaw416c2k
2022-05-18T04:10:33.3372187Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpaw416c2k/_remote_module_non_scriptable.py
2022-05-18T04:10:33.6090517Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:10:33.6100955Z 
2022-05-18T04:10:33.6101560Z Running tests...
2022-05-18T04:10:33.6102202Z ----------------------------------------------------------------------
2022-05-18T04:10:33.9580014Z   test_graph_for_py_nested_remote_call_itself_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24436
2022-05-18T04:10:33.9605493Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24437
2022-05-18T04:10:33.9629845Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24438
2022-05-18T04:10:33.9656204Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 24439
2022-05-18T04:10:34.5576804Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa8q00op6
2022-05-18T04:10:34.5578132Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa8q00op6/_remote_module_non_scriptable.py
2022-05-18T04:10:34.5996780Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzp9bd_61
2022-05-18T04:10:34.5997887Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzp9bd_61/_remote_module_non_scriptable.py
2022-05-18T04:10:34.6359615Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3sknt3eo
2022-05-18T04:10:34.6360597Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3sknt3eo/_remote_module_non_scriptable.py
2022-05-18T04:10:34.6463792Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpssdlbtfu
2022-05-18T04:10:34.6464799Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpssdlbtfu/_remote_module_non_scriptable.py
2022-05-18T04:10:34.8209115Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:10:34.8636347Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:10:34.8996058Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:10:34.9076466Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:10:35.2039936Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:10:35.2040835Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:10:35.2142013Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:10:35.2142850Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T04:10:35.2144405Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:10:35.2145605Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:10:35.2146811Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:10:35.2152928Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:10:35.6703236Z ok (2.060s)
2022-05-18T04:10:35.6703498Z 
2022-05-18T04:10:35.6703821Z ----------------------------------------------------------------------
2022-05-18T04:10:35.6704074Z Ran 1 test in 2.060s
2022-05-18T04:10:35.6704192Z 
2022-05-18T04:10:35.6704256Z OK
2022-05-18T04:10:35.6704356Z 
2022-05-18T04:10:35.6704438Z Generating XML reports...
2022-05-18T04:10:35.6746165Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220518041033.xml
2022-05-18T04:10:36.5311918Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppjl991o8
2022-05-18T04:10:36.5313043Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppjl991o8/_remote_module_non_scriptable.py
2022-05-18T04:10:36.7987090Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:10:36.7996510Z 
2022-05-18T04:10:36.7996725Z Running tests...
2022-05-18T04:10:36.7997380Z ----------------------------------------------------------------------
2022-05-18T04:10:37.1409091Z   test_graph_for_py_nested_remote_call_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24679
2022-05-18T04:10:37.1434588Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24680
2022-05-18T04:10:37.1458730Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24681
2022-05-18T04:10:37.1484619Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 24682
2022-05-18T04:10:37.7416325Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_whef728
2022-05-18T04:10:37.7417108Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_whef728/_remote_module_non_scriptable.py
2022-05-18T04:10:37.7734458Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsd5y0cw6
2022-05-18T04:10:37.7735297Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsd5y0cw6/_remote_module_non_scriptable.py
2022-05-18T04:10:37.8018471Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1pjhobue
2022-05-18T04:10:37.8019251Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1pjhobue/_remote_module_non_scriptable.py
2022-05-18T04:10:37.8040782Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpznsxjks9
2022-05-18T04:10:37.8042887Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpznsxjks9/_remote_module_non_scriptable.py
2022-05-18T04:10:38.0043650Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:10:38.0320234Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:10:38.0599562Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:10:38.0649213Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:10:38.2872506Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:10:38.2973511Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:10:38.3075705Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:10:38.3076579Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T04:10:38.3079648Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:10:38.3080847Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:10:38.3081999Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:10:38.3083151Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:10:38.6526291Z ok (1.853s)
2022-05-18T04:10:38.6526474Z 
2022-05-18T04:10:38.6527029Z ----------------------------------------------------------------------
2022-05-18T04:10:38.6527462Z Ran 1 test in 1.853s
2022-05-18T04:10:38.6527599Z 
2022-05-18T04:10:38.6527661Z OK
2022-05-18T04:10:38.6527754Z 
2022-05-18T04:10:38.6527852Z Generating XML reports...
2022-05-18T04:10:38.6563800Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220518041036.xml
2022-05-18T04:10:39.4548023Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp719duca7
2022-05-18T04:10:39.4548656Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp719duca7/_remote_module_non_scriptable.py
2022-05-18T04:10:39.7099664Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:10:39.7109624Z 
2022-05-18T04:10:39.7109714Z Running tests...
2022-05-18T04:10:39.7110565Z ----------------------------------------------------------------------
2022-05-18T04:10:40.0326755Z   test_graph_for_python_call_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24922
2022-05-18T04:10:40.0349183Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24923
2022-05-18T04:10:40.0372497Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24924
2022-05-18T04:10:40.0397205Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 24925
2022-05-18T04:10:40.6747127Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp31yktqhn
2022-05-18T04:10:40.6747893Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp31yktqhn/_remote_module_non_scriptable.py
2022-05-18T04:10:40.6911113Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuq0nlsuw
2022-05-18T04:10:40.6913936Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuq0nlsuw/_remote_module_non_scriptable.py
2022-05-18T04:10:40.7063539Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpguptxuzg
2022-05-18T04:10:40.7064779Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpguptxuzg/_remote_module_non_scriptable.py
2022-05-18T04:10:40.7197762Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsttijj2b
2022-05-18T04:10:40.7198528Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsttijj2b/_remote_module_non_scriptable.py
2022-05-18T04:10:40.9284140Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:10:40.9454074Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:10:40.9605524Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:10:40.9715103Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:10:41.2174289Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:10:41.2273777Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:10:41.2375276Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:10:41.2385531Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:10:41.2386245Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T04:10:41.2387055Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:10:41.2387889Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:10:41.2388710Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:10:41.6441044Z ok (1.933s)
2022-05-18T04:10:41.6441283Z 
2022-05-18T04:10:41.6441794Z ----------------------------------------------------------------------
2022-05-18T04:10:41.6442064Z Ran 1 test in 1.933s
2022-05-18T04:10:41.6442182Z 
2022-05-18T04:10:41.6442479Z OK
2022-05-18T04:10:41.6442571Z 
2022-05-18T04:10:41.6442665Z Generating XML reports...
2022-05-18T04:10:41.6476025Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220518041039.xml
2022-05-18T04:10:42.4507209Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpax326ryv
2022-05-18T04:10:42.4507659Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpax326ryv/_remote_module_non_scriptable.py
2022-05-18T04:10:42.7088432Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:10:42.7098379Z 
2022-05-18T04:10:42.7098503Z Running tests...
2022-05-18T04:10:42.7099087Z ----------------------------------------------------------------------
2022-05-18T04:10:43.0359128Z   test_graph_for_python_remote_call_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25165
2022-05-18T04:10:43.0382381Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25166
2022-05-18T04:10:43.0406251Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25167
2022-05-18T04:10:43.0431204Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 25168
2022-05-18T04:10:43.7478838Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_xt6bbxw
2022-05-18T04:10:43.7479823Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_xt6bbxw/_remote_module_non_scriptable.py
2022-05-18T04:10:43.7857714Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk6lk4u0b
2022-05-18T04:10:43.7858507Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk6lk4u0b/_remote_module_non_scriptable.py
2022-05-18T04:10:43.8174857Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpshj_8i5k
2022-05-18T04:10:43.8175961Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpshj_8i5k/_remote_module_non_scriptable.py
2022-05-18T04:10:43.8308613Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsu0jv7k_
2022-05-18T04:10:43.8309411Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsu0jv7k_/_remote_module_non_scriptable.py
2022-05-18T04:10:44.0009343Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:10:44.0387718Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:10:44.0681340Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:10:44.0843572Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:10:44.3377349Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:10:44.3380483Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:10:44.3477411Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T04:10:44.3478645Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:10:44.3480295Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:10:44.3481659Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:10:44.3483781Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:10:44.3486585Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:10:44.8479712Z ok (2.138s)
2022-05-18T04:10:44.8479988Z 
2022-05-18T04:10:44.8480517Z ----------------------------------------------------------------------
2022-05-18T04:10:44.8481039Z Ran 1 test in 2.138s
2022-05-18T04:10:44.8481156Z 
2022-05-18T04:10:44.8481218Z OK
2022-05-18T04:10:44.8481311Z 
2022-05-18T04:10:44.8481473Z Generating XML reports...
2022-05-18T04:10:44.8515154Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220518041042.xml
2022-05-18T04:10:45.6859993Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphna3unb9
2022-05-18T04:10:45.6860467Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphna3unb9/_remote_module_non_scriptable.py
2022-05-18T04:10:45.9490819Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:10:45.9500770Z 
2022-05-18T04:10:45.9500857Z Running tests...
2022-05-18T04:10:45.9501918Z ----------------------------------------------------------------------
2022-05-18T04:10:46.2836451Z   test_mixed_requires_grad_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25408
2022-05-18T04:10:46.2859476Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25409
2022-05-18T04:10:46.2883680Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25410
2022-05-18T04:10:46.2908036Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 25411
2022-05-18T04:10:46.9728136Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7hf52lnl
2022-05-18T04:10:46.9728904Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7hf52lnl/_remote_module_non_scriptable.py
2022-05-18T04:10:46.9894375Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2vuys2tz
2022-05-18T04:10:46.9895285Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2vuys2tz/_remote_module_non_scriptable.py
2022-05-18T04:10:47.0104987Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvt6nkbuw
2022-05-18T04:10:47.0106230Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvt6nkbuw/_remote_module_non_scriptable.py
2022-05-18T04:10:47.0258638Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpycjne1jh
2022-05-18T04:10:47.0259571Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpycjne1jh/_remote_module_non_scriptable.py
2022-05-18T04:10:47.2234638Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:10:47.2399035Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:10:47.2616665Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:10:47.2789144Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:10:47.8952981Z ok (1.945s)
2022-05-18T04:10:47.8953279Z 
2022-05-18T04:10:47.8953648Z ----------------------------------------------------------------------
2022-05-18T04:10:47.8953915Z Ran 1 test in 1.945s
2022-05-18T04:10:47.8954033Z 
2022-05-18T04:10:47.8954081Z OK
2022-05-18T04:10:47.8954184Z 
2022-05-18T04:10:47.8954277Z Generating XML reports...
2022-05-18T04:10:47.8987998Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220518041045.xml
2022-05-18T04:10:48.7168916Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpewtu0um_
2022-05-18T04:10:48.7169703Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpewtu0um_/_remote_module_non_scriptable.py
2022-05-18T04:10:48.9705525Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:10:48.9714934Z 
2022-05-18T04:10:48.9715065Z Running tests...
2022-05-18T04:10:48.9715655Z ----------------------------------------------------------------------
2022-05-18T04:10:49.2905354Z   test_multiple_backward_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25659
2022-05-18T04:10:49.2929107Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25660
2022-05-18T04:10:49.2952143Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25661
2022-05-18T04:10:49.2976626Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 25662
2022-05-18T04:10:49.9169127Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxciu7__y
2022-05-18T04:10:49.9170495Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxciu7__y/_remote_module_non_scriptable.py
2022-05-18T04:10:49.9240582Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppthdoawi
2022-05-18T04:10:49.9242206Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppthdoawi/_remote_module_non_scriptable.py
2022-05-18T04:10:49.9295104Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmpkbxc3l
2022-05-18T04:10:49.9296813Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmpkbxc3l/_remote_module_non_scriptable.py
2022-05-18T04:10:49.9459742Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9j7xvzs0
2022-05-18T04:10:49.9460760Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9j7xvzs0/_remote_module_non_scriptable.py
2022-05-18T04:10:50.1669271Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:10:50.1750842Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:10:50.1806094Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:10:50.1947859Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:10:51.8040141Z ok (2.832s)
2022-05-18T04:10:51.8040413Z 
2022-05-18T04:10:51.8040906Z ----------------------------------------------------------------------
2022-05-18T04:10:51.8041363Z Ran 1 test in 2.832s
2022-05-18T04:10:51.8041589Z 
2022-05-18T04:10:51.8041699Z OK
2022-05-18T04:10:51.8041862Z 
2022-05-18T04:10:51.8042036Z Generating XML reports...
2022-05-18T04:10:51.8075408Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220518041048.xml
2022-05-18T04:10:52.6187902Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpidxpt2oq
2022-05-18T04:10:52.6188507Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpidxpt2oq/_remote_module_non_scriptable.py
2022-05-18T04:10:52.8773660Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:10:52.8784741Z 
2022-05-18T04:10:52.8784882Z Running tests...
2022-05-18T04:10:52.8785372Z ----------------------------------------------------------------------
2022-05-18T04:10:53.2103966Z   test_nested_backward_accumulate_grads_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25910
2022-05-18T04:10:53.2126824Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25911
2022-05-18T04:10:53.2150472Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25912
2022-05-18T04:10:53.2176862Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 25913
2022-05-18T04:10:53.8399104Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppmg_6w9_
2022-05-18T04:10:53.8399887Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppmg_6w9_/_remote_module_non_scriptable.py
2022-05-18T04:10:53.8690003Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp395or9cq
2022-05-18T04:10:53.8690785Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp395or9cq/_remote_module_non_scriptable.py
2022-05-18T04:10:53.8691670Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprc7iw76r
2022-05-18T04:10:53.8694433Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprc7iw76r/_remote_module_non_scriptable.py
2022-05-18T04:10:53.8696372Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpspwbfw7j
2022-05-18T04:10:53.8699339Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpspwbfw7j/_remote_module_non_scriptable.py
2022-05-18T04:10:54.0906548Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:10:54.1174126Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:10:54.1198219Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:10:54.1208274Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:10:54.6216115Z ok (1.743s)
2022-05-18T04:10:54.6216377Z 
2022-05-18T04:10:54.6216918Z ----------------------------------------------------------------------
2022-05-18T04:10:54.6217357Z Ran 1 test in 1.743s
2022-05-18T04:10:54.6217479Z 
2022-05-18T04:10:54.6217540Z OK
2022-05-18T04:10:54.6217618Z 
2022-05-18T04:10:54.6217710Z Generating XML reports...
2022-05-18T04:10:54.6251749Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220518041052.xml
2022-05-18T04:10:55.4433644Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8jlrz4on
2022-05-18T04:10:55.4434498Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8jlrz4on/_remote_module_non_scriptable.py
2022-05-18T04:10:55.7026679Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:10:55.7036541Z 
2022-05-18T04:10:55.7036644Z Running tests...
2022-05-18T04:10:55.7037073Z ----------------------------------------------------------------------
2022-05-18T04:10:56.0294612Z   test_no_graph_with_tensors_not_require_grad_remote_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26161
2022-05-18T04:10:56.0317574Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26162
2022-05-18T04:10:56.0341284Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26163
2022-05-18T04:10:56.0366279Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 26164
2022-05-18T04:10:56.6846363Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmploj7n7ji
2022-05-18T04:10:56.6848224Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmploj7n7ji/_remote_module_non_scriptable.py
2022-05-18T04:10:56.6927262Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0rdrg3s6
2022-05-18T04:10:56.6928552Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0rdrg3s6/_remote_module_non_scriptable.py
2022-05-18T04:10:56.7522523Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmmbp18c6
2022-05-18T04:10:56.7523352Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmmbp18c6/_remote_module_non_scriptable.py
2022-05-18T04:10:56.7685716Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg4rbyesy
2022-05-18T04:10:56.7687889Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg4rbyesy/_remote_module_non_scriptable.py
2022-05-18T04:10:56.9336163Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:10:56.9399467Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:10:57.0003216Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:10:57.0194514Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:10:57.2375772Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:10:57.2376677Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T04:10:57.2380657Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:10:57.2381467Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:10:57.2382738Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:10:57.2384132Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:10:57.2477081Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:10:57.2478299Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:10:57.6411141Z ok (1.937s)
2022-05-18T04:10:57.6411395Z 
2022-05-18T04:10:57.6411908Z ----------------------------------------------------------------------
2022-05-18T04:10:57.6412164Z Ran 1 test in 1.937s
2022-05-18T04:10:57.6412280Z 
2022-05-18T04:10:57.6412343Z OK
2022-05-18T04:10:57.6412435Z 
2022-05-18T04:10:57.6412533Z Generating XML reports...
2022-05-18T04:10:57.6447769Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220518041055.xml
2022-05-18T04:10:58.4258811Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp54w4bkw3
2022-05-18T04:10:58.4259835Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp54w4bkw3/_remote_module_non_scriptable.py
2022-05-18T04:10:58.6829104Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:10:58.6838441Z 
2022-05-18T04:10:58.6838562Z Running tests...
2022-05-18T04:10:58.6839234Z ----------------------------------------------------------------------
2022-05-18T04:10:59.0063166Z   test_no_graph_with_tensors_not_require_grad_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26404
2022-05-18T04:10:59.0085184Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26405
2022-05-18T04:10:59.0108619Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26406
2022-05-18T04:10:59.0133538Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 26407
2022-05-18T04:10:59.6657592Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl5jqcga3
2022-05-18T04:10:59.6667630Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl5jqcga3/_remote_module_non_scriptable.py
2022-05-18T04:10:59.6728611Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb5no2sk3
2022-05-18T04:10:59.6729271Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpks_r86kp
2022-05-18T04:10:59.6731764Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb5no2sk3/_remote_module_non_scriptable.py
2022-05-18T04:10:59.6732482Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpks_r86kp/_remote_module_non_scriptable.py
2022-05-18T04:10:59.6798836Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9xocl0vg
2022-05-18T04:10:59.6800240Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9xocl0vg/_remote_module_non_scriptable.py
2022-05-18T04:10:59.9226053Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:10:59.9259577Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:10:59.9266260Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:10:59.9349164Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:11:00.1817388Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:11:00.1872872Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:11:00.1873294Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T04:11:00.1873648Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:11:00.1876170Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:11:00.1877367Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:11:00.1879549Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:11:00.1880684Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:11:00.6178637Z ok (1.934s)
2022-05-18T04:11:00.6178881Z 
2022-05-18T04:11:00.6179390Z ----------------------------------------------------------------------
2022-05-18T04:11:00.6179769Z Ran 1 test in 1.934s
2022-05-18T04:11:00.6179883Z 
2022-05-18T04:11:00.6179944Z OK
2022-05-18T04:11:00.6180037Z 
2022-05-18T04:11:00.6180117Z Generating XML reports...
2022-05-18T04:11:00.6214950Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220518041058.xml
2022-05-18T04:11:01.4143094Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp21xprbch
2022-05-18T04:11:01.4143803Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp21xprbch/_remote_module_non_scriptable.py
2022-05-18T04:11:01.6712580Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:11:01.6721825Z 
2022-05-18T04:11:01.6721962Z Running tests...
2022-05-18T04:11:01.6722572Z ----------------------------------------------------------------------
2022-05-18T04:11:01.9947757Z   test_remote_complex_args_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26647
2022-05-18T04:11:01.9969940Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26648
2022-05-18T04:11:01.9993356Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26649
2022-05-18T04:11:02.0018019Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 26650
2022-05-18T04:11:02.5927228Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf3hkiwau
2022-05-18T04:11:02.5928254Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf3hkiwau/_remote_module_non_scriptable.py
2022-05-18T04:11:02.6243323Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpaskmj8yc
2022-05-18T04:11:02.6244033Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpaskmj8yc/_remote_module_non_scriptable.py
2022-05-18T04:11:02.6425890Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn8p94wna
2022-05-18T04:11:02.6427095Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn8p94wna/_remote_module_non_scriptable.py
2022-05-18T04:11:02.6481541Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_p0vfgiw
2022-05-18T04:11:02.6483751Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_p0vfgiw/_remote_module_non_scriptable.py
2022-05-18T04:11:02.8466463Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:11:02.8757498Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:11:02.8970442Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:11:02.8982531Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:11:03.4057785Z ok (1.733s)
2022-05-18T04:11:03.4057967Z 
2022-05-18T04:11:03.4058451Z ----------------------------------------------------------------------
2022-05-18T04:11:03.4058908Z Ran 1 test in 1.734s
2022-05-18T04:11:03.4059046Z 
2022-05-18T04:11:03.4059109Z OK
2022-05-18T04:11:03.4059201Z 
2022-05-18T04:11:03.4059296Z Generating XML reports...
2022-05-18T04:11:03.4095368Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220518041101.xml
2022-05-18T04:11:04.2030508Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7nupy39u
2022-05-18T04:11:04.2031352Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7nupy39u/_remote_module_non_scriptable.py
2022-05-18T04:11:04.4596469Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:11:04.4605819Z 
2022-05-18T04:11:04.4605928Z Running tests...
2022-05-18T04:11:04.4606520Z ----------------------------------------------------------------------
2022-05-18T04:11:04.7781785Z   test_rpc_complex_args_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26890
2022-05-18T04:11:04.7803865Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26891
2022-05-18T04:11:04.7827199Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26892
2022-05-18T04:11:04.7851974Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 26893
2022-05-18T04:11:05.4043087Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7iy6pkr6
2022-05-18T04:11:05.4043894Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7iy6pkr6/_remote_module_non_scriptable.py
2022-05-18T04:11:05.4121793Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8hhbpzdy
2022-05-18T04:11:05.4122885Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8hhbpzdy/_remote_module_non_scriptable.py
2022-05-18T04:11:05.4127129Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi_j6rk1m
2022-05-18T04:11:05.4129252Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi_j6rk1m/_remote_module_non_scriptable.py
2022-05-18T04:11:05.4302827Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6r5lpxnk
2022-05-18T04:11:05.4303955Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6r5lpxnk/_remote_module_non_scriptable.py
2022-05-18T04:11:05.6547469Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:11:05.6612739Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:11:05.6631526Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:11:05.6784528Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:11:06.4897549Z ok (2.029s)
2022-05-18T04:11:06.4897825Z 
2022-05-18T04:11:06.4898338Z ----------------------------------------------------------------------
2022-05-18T04:11:06.4898692Z Ran 1 test in 2.029s
2022-05-18T04:11:06.4898866Z 
2022-05-18T04:11:06.4898930Z OK
2022-05-18T04:11:06.4899065Z 
2022-05-18T04:11:06.4899209Z Generating XML reports...
2022-05-18T04:11:06.4933496Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220518041104.xml
2022-05-18T04:11:07.2654339Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfn8arxx7
2022-05-18T04:11:07.2655255Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfn8arxx7/_remote_module_non_scriptable.py
2022-05-18T04:11:07.5204537Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:11:07.5214791Z 
2022-05-18T04:11:07.5215103Z Running tests...
2022-05-18T04:11:07.5215748Z ----------------------------------------------------------------------
2022-05-18T04:11:07.8423723Z   test_trainer_ps_sparse (__main__.TensorPipeTensorPipeAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27133
2022-05-18T04:11:07.8446607Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27134
2022-05-18T04:11:07.8470321Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27135
2022-05-18T04:11:07.8495203Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 27136
2022-05-18T04:11:08.4524731Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb7l1wup0
2022-05-18T04:11:08.4525515Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb7l1wup0/_remote_module_non_scriptable.py
2022-05-18T04:11:08.4614367Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8oxu1bk5
2022-05-18T04:11:08.4615274Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8oxu1bk5/_remote_module_non_scriptable.py
2022-05-18T04:11:08.4730628Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdj0izvt5
2022-05-18T04:11:08.4731371Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdj0izvt5/_remote_module_non_scriptable.py
2022-05-18T04:11:08.4958133Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpscvf1c6n
2022-05-18T04:11:08.4960536Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpscvf1c6n/_remote_module_non_scriptable.py
2022-05-18T04:11:08.7035148Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:11:08.7111473Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:11:08.7239370Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:11:08.7444831Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:11:09.4541656Z ok (1.932s)
2022-05-18T04:11:09.4541932Z 
2022-05-18T04:11:09.4542472Z ----------------------------------------------------------------------
2022-05-18T04:11:09.4542740Z Ran 1 test in 1.933s
2022-05-18T04:11:09.4542856Z 
2022-05-18T04:11:09.4543020Z OK
2022-05-18T04:11:09.4543116Z 
2022-05-18T04:11:09.4543216Z Generating XML reports...
2022-05-18T04:11:09.4576885Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220518041107.xml
2022-05-18T04:11:10.2624092Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprm2p12bj
2022-05-18T04:11:10.2624702Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprm2p12bj/_remote_module_non_scriptable.py
2022-05-18T04:11:10.5190921Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:11:10.5200009Z 
2022-05-18T04:11:10.5200135Z Running tests...
2022-05-18T04:11:10.5200705Z ----------------------------------------------------------------------
2022-05-18T04:11:10.8447070Z   test_builtin_remote_ret_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27396
2022-05-18T04:11:10.8470497Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27397
2022-05-18T04:11:10.8494629Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27398
2022-05-18T04:11:10.8520449Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 27399
2022-05-18T04:11:11.4556254Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpddx9_y7e
2022-05-18T04:11:11.4557447Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpddx9_y7e/_remote_module_non_scriptable.py
2022-05-18T04:11:11.4721285Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf1_8d9d3
2022-05-18T04:11:11.4722399Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf1_8d9d3/_remote_module_non_scriptable.py
2022-05-18T04:11:11.5021165Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp71h4ghhd
2022-05-18T04:11:11.5021909Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp71h4ghhd/_remote_module_non_scriptable.py
2022-05-18T04:11:11.5037982Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1eu11dkh
2022-05-18T04:11:11.5039842Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1eu11dkh/_remote_module_non_scriptable.py
2022-05-18T04:11:11.7080639Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:11:11.7240759Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:11:11.7499337Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:11:11.7547448Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:11:12.2559633Z ok (1.736s)
2022-05-18T04:11:12.2559888Z 
2022-05-18T04:11:12.2560392Z ----------------------------------------------------------------------
2022-05-18T04:11:12.2560664Z Ran 1 test in 1.736s
2022-05-18T04:11:12.2560765Z 
2022-05-18T04:11:12.2560826Z OK
2022-05-18T04:11:12.2560920Z 
2022-05-18T04:11:12.2561016Z Generating XML reports...
2022-05-18T04:11:12.2595428Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220518041110.xml
2022-05-18T04:11:13.0477407Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplo71217_
2022-05-18T04:11:13.0478191Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplo71217_/_remote_module_non_scriptable.py
2022-05-18T04:11:13.3032668Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:11:13.3042672Z 
2022-05-18T04:11:13.3042943Z Running tests...
2022-05-18T04:11:13.3043641Z ----------------------------------------------------------------------
2022-05-18T04:11:13.6243873Z   test_builtin_remote_self_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27627
2022-05-18T04:11:13.6267654Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27628
2022-05-18T04:11:13.6290848Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27629
2022-05-18T04:11:13.6316772Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 27630
2022-05-18T04:11:14.2460823Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3yq_r2c9
2022-05-18T04:11:14.2463619Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3yq_r2c9/_remote_module_non_scriptable.py
2022-05-18T04:11:14.2619514Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl8vu9pn6
2022-05-18T04:11:14.2620486Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl8vu9pn6/_remote_module_non_scriptable.py
2022-05-18T04:11:14.2676365Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1_r8xp0i
2022-05-18T04:11:14.2677934Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1_r8xp0i/_remote_module_non_scriptable.py
2022-05-18T04:11:14.2808942Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8te8l6b7
2022-05-18T04:11:14.2809959Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8te8l6b7/_remote_module_non_scriptable.py
2022-05-18T04:11:14.4972090Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:11:14.5135088Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:11:14.5164801Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:11:14.5268119Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:11:15.1359385Z ok (1.831s)
2022-05-18T04:11:15.1359652Z 
2022-05-18T04:11:15.1360163Z ----------------------------------------------------------------------
2022-05-18T04:11:15.1360633Z Ran 1 test in 1.832s
2022-05-18T04:11:15.1360849Z 
2022-05-18T04:11:15.1360958Z OK
2022-05-18T04:11:15.1361130Z 
2022-05-18T04:11:15.1361309Z Generating XML reports...
2022-05-18T04:11:15.1397014Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220518041113.xml
2022-05-18T04:11:15.9163709Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphcyyjgd9
2022-05-18T04:11:15.9164911Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphcyyjgd9/_remote_module_non_scriptable.py
2022-05-18T04:11:16.1689984Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:11:16.1699470Z 
2022-05-18T04:11:16.1699573Z Running tests...
2022-05-18T04:11:16.1700685Z ----------------------------------------------------------------------
2022-05-18T04:11:16.4880553Z   test_infer_backend_from_options (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27858
2022-05-18T04:11:16.4903546Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27859
2022-05-18T04:11:16.4926624Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27860
2022-05-18T04:11:16.4951503Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 27861
2022-05-18T04:11:17.0909703Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_2b82wg9
2022-05-18T04:11:17.0910491Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_2b82wg9/_remote_module_non_scriptable.py
2022-05-18T04:11:17.1116876Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa49w4map
2022-05-18T04:11:17.1117631Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa49w4map/_remote_module_non_scriptable.py
2022-05-18T04:11:17.1189200Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwhaldfzy
2022-05-18T04:11:17.1190719Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwhaldfzy/_remote_module_non_scriptable.py
2022-05-18T04:11:17.1323890Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp990ebxmf
2022-05-18T04:11:17.1325363Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp990ebxmf/_remote_module_non_scriptable.py
2022-05-18T04:11:17.3397008Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:11:17.3588166Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:11:17.3694306Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:11:17.3820906Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:11:17.7545030Z [W tensorpipe_agent.cpp:728] RPC agent for worker2 encountered error when reading incoming request from worker0: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2022-05-18T04:11:17.7546081Z [W tensorpipe_agent.cpp:728] RPC agent for worker3 encountered error when reading incoming request from worker0: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2022-05-18T04:11:17.7547180Z [W tensorpipe_agent.cpp:728] RPC agent for worker1 encountered error when reading incoming request from worker0: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2022-05-18T04:11:17.8992983Z ok (1.729s)
2022-05-18T04:11:17.8993235Z 
2022-05-18T04:11:17.8993686Z ----------------------------------------------------------------------
2022-05-18T04:11:17.8994144Z Ran 1 test in 1.729s
2022-05-18T04:11:17.8994365Z 
2022-05-18T04:11:17.8994465Z OK
2022-05-18T04:11:17.8994578Z 
2022-05-18T04:11:17.8994680Z Generating XML reports...
2022-05-18T04:11:17.9027699Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220518041116.xml
2022-05-18T04:11:18.6725640Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5egrpeg0
2022-05-18T04:11:18.6726403Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5egrpeg0/_remote_module_non_scriptable.py
2022-05-18T04:11:18.9257378Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:11:18.9267765Z 
2022-05-18T04:11:18.9268088Z Running tests...
2022-05-18T04:11:18.9268731Z ----------------------------------------------------------------------
2022-05-18T04:11:19.2432305Z   test_meta_multiple_tensors (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28077
2022-05-18T04:11:19.2454705Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28078
2022-05-18T04:11:19.2478109Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28079
2022-05-18T04:11:19.2502836Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 28080
2022-05-18T04:11:19.8971671Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe4oja95o
2022-05-18T04:11:19.8972782Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe4oja95o/_remote_module_non_scriptable.py
2022-05-18T04:11:19.9252171Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm9x9_l5y
2022-05-18T04:11:19.9252925Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm9x9_l5y/_remote_module_non_scriptable.py
2022-05-18T04:11:19.9271983Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprx6qbp4k
2022-05-18T04:11:19.9273413Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprx6qbp4k/_remote_module_non_scriptable.py
2022-05-18T04:11:19.9447711Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq31y00cl
2022-05-18T04:11:19.9449700Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq31y00cl/_remote_module_non_scriptable.py
2022-05-18T04:11:20.1461316Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:11:20.1767073Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:11:20.1872666Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:11:20.2047422Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:11:20.7545132Z ok (1.827s)
2022-05-18T04:11:20.7545377Z 
2022-05-18T04:11:20.7545827Z ----------------------------------------------------------------------
2022-05-18T04:11:20.7546090Z Ran 1 test in 1.828s
2022-05-18T04:11:20.7546205Z 
2022-05-18T04:11:20.7546267Z OK
2022-05-18T04:11:20.7546359Z 
2022-05-18T04:11:20.7546481Z Generating XML reports...
2022-05-18T04:11:20.7580370Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220518041118.xml
2022-05-18T04:11:21.5248512Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpct3fwy5w
2022-05-18T04:11:21.5249250Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpct3fwy5w/_remote_module_non_scriptable.py
2022-05-18T04:11:21.7777865Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:11:21.7787328Z 
2022-05-18T04:11:21.7787511Z Running tests...
2022-05-18T04:11:21.7788116Z ----------------------------------------------------------------------
2022-05-18T04:11:22.0951258Z   test_meta_one_tensor (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28296
2022-05-18T04:11:22.0974915Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28297
2022-05-18T04:11:22.0998025Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28298
2022-05-18T04:11:22.1023055Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 28299
2022-05-18T04:11:22.7310344Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplkbfd9xr
2022-05-18T04:11:22.7311646Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplkbfd9xr/_remote_module_non_scriptable.py
2022-05-18T04:11:22.7357005Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqvvhbok7
2022-05-18T04:11:22.7357920Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqvvhbok7/_remote_module_non_scriptable.py
2022-05-18T04:11:22.7431757Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpasjktq1t
2022-05-18T04:11:22.7433319Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpasjktq1t/_remote_module_non_scriptable.py
2022-05-18T04:11:22.7675117Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7_09x6yx
2022-05-18T04:11:22.7676256Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7_09x6yx/_remote_module_non_scriptable.py
2022-05-18T04:11:22.9807906Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:11:22.9836659Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:11:22.9902567Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:11:23.0142297Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:11:23.5063581Z ok (1.727s)
2022-05-18T04:11:23.5063879Z 
2022-05-18T04:11:23.5064367Z ----------------------------------------------------------------------
2022-05-18T04:11:23.5064620Z Ran 1 test in 1.728s
2022-05-18T04:11:23.5064739Z 
2022-05-18T04:11:23.5064801Z OK
2022-05-18T04:11:23.5064895Z 
2022-05-18T04:11:23.5064989Z Generating XML reports...
2022-05-18T04:11:23.5099032Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220518041121.xml
2022-05-18T04:11:24.2787206Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm7rz9vnv
2022-05-18T04:11:24.2787947Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm7rz9vnv/_remote_module_non_scriptable.py
2022-05-18T04:11:24.5312784Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:11:24.5322862Z 
2022-05-18T04:11:24.5323328Z Running tests...
2022-05-18T04:11:24.5323747Z ----------------------------------------------------------------------
2022-05-18T04:11:24.8477973Z   test_meta_one_tensor_rref (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28515
2022-05-18T04:11:24.8501240Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28516
2022-05-18T04:11:24.8525289Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28517
2022-05-18T04:11:24.8549670Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 28518
2022-05-18T04:11:25.4998604Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc1lk43t4
2022-05-18T04:11:25.4999628Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc1lk43t4/_remote_module_non_scriptable.py
2022-05-18T04:11:25.5129734Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj4tw25ow
2022-05-18T04:11:25.5130553Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj4tw25ow/_remote_module_non_scriptable.py
2022-05-18T04:11:25.5258968Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv65o7mgw
2022-05-18T04:11:25.5259703Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv65o7mgw/_remote_module_non_scriptable.py
2022-05-18T04:11:25.5389336Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd5go1m4x
2022-05-18T04:11:25.5390926Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd5go1m4x/_remote_module_non_scriptable.py
2022-05-18T04:11:25.7480586Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:11:25.7612496Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:11:25.7728075Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:11:25.7896259Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:11:26.3592327Z ok (1.827s)
2022-05-18T04:11:26.3592556Z 
2022-05-18T04:11:26.3592996Z ----------------------------------------------------------------------
2022-05-18T04:11:26.3593400Z Ran 1 test in 1.827s
2022-05-18T04:11:26.3593570Z 
2022-05-18T04:11:26.3593664Z OK
2022-05-18T04:11:26.3593804Z 
2022-05-18T04:11:26.3593947Z Generating XML reports...
2022-05-18T04:11:26.3628537Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220518041124.xml
2022-05-18T04:11:27.1313624Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzsyushrr
2022-05-18T04:11:27.1314219Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzsyushrr/_remote_module_non_scriptable.py
2022-05-18T04:11:27.3844887Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:11:27.3853982Z 
2022-05-18T04:11:27.3854125Z Running tests...
2022-05-18T04:11:27.3854517Z ----------------------------------------------------------------------
2022-05-18T04:11:27.7017083Z   test_mismatched_type_for_options (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28734
2022-05-18T04:11:27.7040481Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28735
2022-05-18T04:11:27.7064174Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28736
2022-05-18T04:11:27.7088242Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 28737
2022-05-18T04:11:28.4125418Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7d0_uhj3
2022-05-18T04:11:28.4126422Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppddya_u5
2022-05-18T04:11:28.4126848Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppddya_u5/_remote_module_non_scriptable.py
2022-05-18T04:11:28.4127258Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7d0_uhj3/_remote_module_non_scriptable.py
2022-05-18T04:11:28.4218221Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprhecqtmd
2022-05-18T04:11:28.4219830Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprhecqtmd/_remote_module_non_scriptable.py
2022-05-18T04:11:28.4336217Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmparyewad2
2022-05-18T04:11:28.4338494Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmparyewad2/_remote_module_non_scriptable.py
2022-05-18T04:11:28.6703710Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:11:28.6852993Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:11:28.6972845Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:11:28.6992986Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:11:28.9124751Z ok (1.527s)
2022-05-18T04:11:28.9125010Z 
2022-05-18T04:11:28.9125545Z ----------------------------------------------------------------------
2022-05-18T04:11:28.9125834Z Ran 1 test in 1.527s
2022-05-18T04:11:28.9125951Z 
2022-05-18T04:11:28.9125999Z OK
2022-05-18T04:11:28.9126090Z 
2022-05-18T04:11:28.9126184Z Generating XML reports...
2022-05-18T04:11:28.9161438Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220518041127.xml
2022-05-18T04:11:29.6702850Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpceh4vp8i
2022-05-18T04:11:29.6703779Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpceh4vp8i/_remote_module_non_scriptable.py
2022-05-18T04:11:29.9269141Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:11:29.9279210Z 
2022-05-18T04:11:29.9279335Z Running tests...
2022-05-18T04:11:29.9279914Z ----------------------------------------------------------------------
2022-05-18T04:11:30.2482321Z   test_multi_builtin_remote_ret_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28789
2022-05-18T04:11:30.2506501Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28790
2022-05-18T04:11:30.2530224Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28791
2022-05-18T04:11:30.2555063Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 28792
2022-05-18T04:11:30.8722152Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf33owwaj
2022-05-18T04:11:30.8724700Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf33owwaj/_remote_module_non_scriptable.py
2022-05-18T04:11:30.8951793Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx0i7me61
2022-05-18T04:11:30.8953239Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx0i7me61/_remote_module_non_scriptable.py
2022-05-18T04:11:30.9005069Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptzg0nscc
2022-05-18T04:11:30.9005995Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg0mdw8o6
2022-05-18T04:11:30.9007180Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptzg0nscc/_remote_module_non_scriptable.py
2022-05-18T04:11:30.9008118Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg0mdw8o6/_remote_module_non_scriptable.py
2022-05-18T04:11:31.1347217Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:11:31.1437315Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:11:31.1496320Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:11:31.1533538Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:11:31.7597880Z ok (1.832s)
2022-05-18T04:11:31.7598147Z 
2022-05-18T04:11:31.7598637Z ----------------------------------------------------------------------
2022-05-18T04:11:31.7598911Z Ran 1 test in 1.832s
2022-05-18T04:11:31.7599026Z 
2022-05-18T04:11:31.7599089Z OK
2022-05-18T04:11:31.7599181Z 
2022-05-18T04:11:31.7599277Z Generating XML reports...
2022-05-18T04:11:31.7632436Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220518041129.xml
2022-05-18T04:11:32.5296994Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcyaoufx9
2022-05-18T04:11:32.5297946Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcyaoufx9/_remote_module_non_scriptable.py
2022-05-18T04:11:32.7832038Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:11:32.7841262Z 
2022-05-18T04:11:32.7841583Z Running tests...
2022-05-18T04:11:32.7842181Z ----------------------------------------------------------------------
2022-05-18T04:11:33.1038568Z   test_multi_py_udf_remote_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29020
2022-05-18T04:11:33.1061086Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29021
2022-05-18T04:11:33.1085989Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29022
2022-05-18T04:11:33.1118323Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 29023
2022-05-18T04:11:33.7319584Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmv_uhman
2022-05-18T04:11:33.7320783Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmv_uhman/_remote_module_non_scriptable.py
2022-05-18T04:11:33.7341274Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp311kpdgz
2022-05-18T04:11:33.7343348Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp311kpdgz/_remote_module_non_scriptable.py
2022-05-18T04:11:33.7379676Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp31p1kgzd
2022-05-18T04:11:33.7381330Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp31p1kgzd/_remote_module_non_scriptable.py
2022-05-18T04:11:33.7705516Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcgrb6861
2022-05-18T04:11:33.7706327Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcgrb6861/_remote_module_non_scriptable.py
2022-05-18T04:11:33.9805191Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:11:33.9817731Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:11:33.9888062Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:11:34.0180143Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:11:34.5157703Z ok (1.731s)
2022-05-18T04:11:34.5157918Z 
2022-05-18T04:11:34.5158369Z ----------------------------------------------------------------------
2022-05-18T04:11:34.5158796Z Ran 1 test in 1.732s
2022-05-18T04:11:34.5158970Z 
2022-05-18T04:11:34.5159060Z OK
2022-05-18T04:11:34.5159200Z 
2022-05-18T04:11:34.5159348Z Generating XML reports...
2022-05-18T04:11:34.5195718Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220518041132.xml
2022-05-18T04:11:35.2909057Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa34gjjrn
2022-05-18T04:11:35.2909983Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa34gjjrn/_remote_module_non_scriptable.py
2022-05-18T04:11:35.5445433Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:11:35.5455145Z 
2022-05-18T04:11:35.5455643Z Running tests...
2022-05-18T04:11:35.5456309Z ----------------------------------------------------------------------
2022-05-18T04:11:35.8603728Z   test_multi_rpc_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29251
2022-05-18T04:11:35.8626076Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29252
2022-05-18T04:11:35.8649307Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29253
2022-05-18T04:11:35.8673769Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 29254
2022-05-18T04:11:36.5370846Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjw6s73fy
2022-05-18T04:11:36.5371848Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjw6s73fy/_remote_module_non_scriptable.py
2022-05-18T04:11:36.5711194Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4cmxfluq
2022-05-18T04:11:36.5712093Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4cmxfluq/_remote_module_non_scriptable.py
2022-05-18T04:11:36.5747993Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp53pfq1ws
2022-05-18T04:11:36.5749230Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp53pfq1ws/_remote_module_non_scriptable.py
2022-05-18T04:11:36.6162411Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmz02fxg2
2022-05-18T04:11:36.6163240Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmz02fxg2/_remote_module_non_scriptable.py
2022-05-18T04:11:36.7874274Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:11:36.8201665Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:11:36.8209747Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:11:36.8626942Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:11:37.7722018Z ok (2.226s)
2022-05-18T04:11:37.7722328Z 
2022-05-18T04:11:37.7722801Z ----------------------------------------------------------------------
2022-05-18T04:11:37.7723075Z Ran 1 test in 2.227s
2022-05-18T04:11:37.7723197Z 
2022-05-18T04:11:37.7723260Z OK
2022-05-18T04:11:37.7723340Z 
2022-05-18T04:11:37.7723434Z Generating XML reports...
2022-05-18T04:11:37.7756996Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220518041135.xml
2022-05-18T04:11:38.5374254Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzfilnwb8
2022-05-18T04:11:38.5375044Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzfilnwb8/_remote_module_non_scriptable.py
2022-05-18T04:11:38.7900749Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:11:38.7911112Z 
2022-05-18T04:11:38.7911522Z Running tests...
2022-05-18T04:11:38.7911914Z ----------------------------------------------------------------------
2022-05-18T04:11:39.1052387Z   test_my_parameter_server_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29482
2022-05-18T04:11:39.1075490Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29483
2022-05-18T04:11:39.1098271Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29484
2022-05-18T04:11:39.1122267Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 29485
2022-05-18T04:11:39.7047726Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb7dc88ck
2022-05-18T04:11:39.7048664Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb7dc88ck/_remote_module_non_scriptable.py
2022-05-18T04:11:39.7093672Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqzlz56ff
2022-05-18T04:11:39.7094858Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqzlz56ff/_remote_module_non_scriptable.py
2022-05-18T04:11:39.7333114Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6n7yv8qg
2022-05-18T04:11:39.7333870Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6n7yv8qg/_remote_module_non_scriptable.py
2022-05-18T04:11:39.7420091Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqkx1po21
2022-05-18T04:11:39.7421462Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqkx1po21/_remote_module_non_scriptable.py
2022-05-18T04:11:39.9530864Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:11:39.9562684Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:11:39.9828694Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:11:39.9875759Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:11:40.6164355Z ok (1.825s)
2022-05-18T04:11:40.6164567Z 
2022-05-18T04:11:40.6165233Z ----------------------------------------------------------------------
2022-05-18T04:11:40.6165698Z Ran 1 test in 1.825s
2022-05-18T04:11:40.6165899Z 
2022-05-18T04:11:40.6165991Z OK
2022-05-18T04:11:40.6166102Z 
2022-05-18T04:11:40.6166186Z Generating XML reports...
2022-05-18T04:11:40.6200324Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220518041138.xml
2022-05-18T04:11:41.3857366Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxd45x6un
2022-05-18T04:11:41.3858437Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxd45x6un/_remote_module_non_scriptable.py
2022-05-18T04:11:41.6385326Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:11:41.6395429Z 
2022-05-18T04:11:41.6395823Z Running tests...
2022-05-18T04:11:41.6396224Z ----------------------------------------------------------------------
2022-05-18T04:11:41.9536767Z   test_nested_remote_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29737
2022-05-18T04:11:41.9558971Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29738
2022-05-18T04:11:41.9581947Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29739
2022-05-18T04:11:41.9606132Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 29740
2022-05-18T04:11:42.6164465Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq7prodlx
2022-05-18T04:11:42.6165482Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq7prodlx/_remote_module_non_scriptable.py
2022-05-18T04:11:42.6184372Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnni01iwh
2022-05-18T04:11:42.6185889Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnni01iwh/_remote_module_non_scriptable.py
2022-05-18T04:11:42.6215082Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzt4upwpf
2022-05-18T04:11:42.6217006Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzt4upwpf/_remote_module_non_scriptable.py
2022-05-18T04:11:42.6412581Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpykri1ioz
2022-05-18T04:11:42.6413321Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpykri1ioz/_remote_module_non_scriptable.py
2022-05-18T04:11:42.8648143Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:11:42.8652836Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:11:42.8705516Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:11:42.8887429Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:11:43.3646073Z ok (1.725s)
2022-05-18T04:11:43.3646380Z 
2022-05-18T04:11:43.3646831Z ----------------------------------------------------------------------
2022-05-18T04:11:43.3647253Z Ran 1 test in 1.725s
2022-05-18T04:11:43.3647434Z 
2022-05-18T04:11:43.3647529Z OK
2022-05-18T04:11:43.3647672Z 
2022-05-18T04:11:43.3647807Z Generating XML reports...
2022-05-18T04:11:43.3681952Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220518041141.xml
2022-05-18T04:11:44.1296628Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyfloxp5t
2022-05-18T04:11:44.1297653Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyfloxp5t/_remote_module_non_scriptable.py
2022-05-18T04:11:44.3810362Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:11:44.3819822Z 
2022-05-18T04:11:44.3819938Z Running tests...
2022-05-18T04:11:44.3820386Z ----------------------------------------------------------------------
2022-05-18T04:11:44.6964978Z   test_nested_rpc_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29980
2022-05-18T04:11:44.6988425Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29981
2022-05-18T04:11:44.7011426Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29982
2022-05-18T04:11:44.7035653Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 29983
2022-05-18T04:11:45.2970818Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp6m0ctvr
2022-05-18T04:11:45.2971603Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp6m0ctvr/_remote_module_non_scriptable.py
2022-05-18T04:11:45.3353162Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4x3aukwp
2022-05-18T04:11:45.3353955Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4x3aukwp/_remote_module_non_scriptable.py
2022-05-18T04:11:45.3564373Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc41p8y5n
2022-05-18T04:11:45.3565302Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc41p8y5n/_remote_module_non_scriptable.py
2022-05-18T04:11:45.3735327Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpghljl373
2022-05-18T04:11:45.3737825Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpghljl373/_remote_module_non_scriptable.py
2022-05-18T04:11:45.5446071Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:11:45.5853938Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:11:45.6053395Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:11:45.6231469Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:11:46.2077684Z ok (1.825s)
2022-05-18T04:11:46.2077856Z 
2022-05-18T04:11:46.2078361Z ----------------------------------------------------------------------
2022-05-18T04:11:46.2078790Z Ran 1 test in 1.826s
2022-05-18T04:11:46.2078915Z 
2022-05-18T04:11:46.2078982Z OK
2022-05-18T04:11:46.2079077Z 
2022-05-18T04:11:46.2079172Z Generating XML reports...
2022-05-18T04:11:46.2113095Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220518041144.xml
2022-05-18T04:11:46.9846857Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8sun8vy_
2022-05-18T04:11:46.9847720Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8sun8vy_/_remote_module_non_scriptable.py
2022-05-18T04:11:47.2373040Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:11:47.2382438Z 
2022-05-18T04:11:47.2382572Z Running tests...
2022-05-18T04:11:47.2383091Z ----------------------------------------------------------------------
2022-05-18T04:11:47.5543883Z   test_nested_rref_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30223
2022-05-18T04:11:47.5568905Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30224
2022-05-18T04:11:47.5591579Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30225
2022-05-18T04:11:47.5616189Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 30226
2022-05-18T04:11:48.2068464Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe76q43sm
2022-05-18T04:11:48.2069600Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe76q43sm/_remote_module_non_scriptable.py
2022-05-18T04:11:48.2348192Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk4zkozq3
2022-05-18T04:11:48.2349204Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk4zkozq3/_remote_module_non_scriptable.py
2022-05-18T04:11:48.2764425Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4ue0hhv6
2022-05-18T04:11:48.2765210Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4ue0hhv6/_remote_module_non_scriptable.py
2022-05-18T04:11:48.2870323Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfe26u3wb
2022-05-18T04:11:48.2870994Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfe26u3wb/_remote_module_non_scriptable.py
2022-05-18T04:11:48.4693822Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:11:48.4953759Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:11:48.5385588Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:11:48.5451003Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:11:49.1661489Z ok (1.928s)
2022-05-18T04:11:49.1661751Z 
2022-05-18T04:11:49.1662250Z ----------------------------------------------------------------------
2022-05-18T04:11:49.1662512Z Ran 1 test in 1.928s
2022-05-18T04:11:49.1662629Z 
2022-05-18T04:11:49.1662690Z OK
2022-05-18T04:11:49.1662784Z 
2022-05-18T04:11:49.1662986Z Generating XML reports...
2022-05-18T04:11:49.1697252Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220518041147.xml
2022-05-18T04:11:50.0085549Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgdde5upy
2022-05-18T04:11:50.0086508Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgdde5upy/_remote_module_non_scriptable.py
2022-05-18T04:11:50.2647495Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:11:50.2657131Z 
2022-05-18T04:11:50.2657355Z Running tests...
2022-05-18T04:11:50.2657973Z ----------------------------------------------------------------------
2022-05-18T04:11:50.5905670Z   test_nested_rref_stress_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30466
2022-05-18T04:11:50.5928428Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30467
2022-05-18T04:11:50.5951974Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30468
2022-05-18T04:11:50.5976525Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 30469
2022-05-18T04:11:51.2561443Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb14cyr3l
2022-05-18T04:11:51.2562247Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb14cyr3l/_remote_module_non_scriptable.py
2022-05-18T04:11:51.2832154Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_xql__q9
2022-05-18T04:11:51.2833093Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_xql__q9/_remote_module_non_scriptable.py
2022-05-18T04:11:51.2973815Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdcoqamo8
2022-05-18T04:11:51.2974648Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdcoqamo8/_remote_module_non_scriptable.py
2022-05-18T04:11:51.3003145Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppvono1w2
2022-05-18T04:11:51.3004422Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppvono1w2/_remote_module_non_scriptable.py
2022-05-18T04:11:51.5048361Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:11:51.5313831Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:11:51.5444405Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:11:51.5463753Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:11:52.2019941Z ok (1.936s)
2022-05-18T04:11:52.2020158Z 
2022-05-18T04:11:52.2020675Z ----------------------------------------------------------------------
2022-05-18T04:11:52.2021109Z Ran 1 test in 1.936s
2022-05-18T04:11:52.2021234Z 
2022-05-18T04:11:52.2021298Z OK
2022-05-18T04:11:52.2021394Z 
2022-05-18T04:11:52.2021490Z Generating XML reports...
2022-05-18T04:11:52.2059105Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220518041150.xml
2022-05-18T04:11:52.9999831Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6igsx9jf
2022-05-18T04:11:53.0000870Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6igsx9jf/_remote_module_non_scriptable.py
2022-05-18T04:11:53.2569724Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:11:53.2579105Z 
2022-05-18T04:11:53.2579202Z Running tests...
2022-05-18T04:11:53.2580183Z ----------------------------------------------------------------------
2022-05-18T04:11:53.5840285Z   test_op_with_invalid_args (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30886
2022-05-18T04:11:53.5863713Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30887
2022-05-18T04:11:53.5887761Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30888
2022-05-18T04:11:53.5912362Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 30889
2022-05-18T04:11:54.2371036Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjph43yrl
2022-05-18T04:11:54.2371951Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjph43yrl/_remote_module_non_scriptable.py
2022-05-18T04:11:54.2467639Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5txjz5da
2022-05-18T04:11:54.2468541Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5txjz5da/_remote_module_non_scriptable.py
2022-05-18T04:11:54.2509150Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2b6pkgg3
2022-05-18T04:11:54.2510174Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2b6pkgg3/_remote_module_non_scriptable.py
2022-05-18T04:11:54.2683782Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptkanj9o_
2022-05-18T04:11:54.2684552Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptkanj9o_/_remote_module_non_scriptable.py
2022-05-18T04:11:54.4952601Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:11:54.5045046Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:11:54.5066528Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:11:54.5279835Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:11:54.9954794Z ok (1.737s)
2022-05-18T04:11:54.9955025Z 
2022-05-18T04:11:54.9955515Z ----------------------------------------------------------------------
2022-05-18T04:11:54.9955982Z Ran 1 test in 1.737s
2022-05-18T04:11:54.9956142Z 
2022-05-18T04:11:54.9956207Z OK
2022-05-18T04:11:54.9956337Z 
2022-05-18T04:11:54.9956434Z Generating XML reports...
2022-05-18T04:11:54.9989747Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220518041153.xml
2022-05-18T04:11:55.8382669Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8rori7ee
2022-05-18T04:11:55.8383335Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8rori7ee/_remote_module_non_scriptable.py
2022-05-18T04:11:56.0972975Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:11:56.0983377Z 
2022-05-18T04:11:56.0983661Z Running tests...
2022-05-18T04:11:56.0984076Z ----------------------------------------------------------------------
2022-05-18T04:11:56.4321295Z   test_py_rpc_rref_args_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31105
2022-05-18T04:11:56.4344522Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31106
2022-05-18T04:11:56.4367986Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31107
2022-05-18T04:11:56.4393062Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 31108
2022-05-18T04:11:57.0309309Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp97sksb7l
2022-05-18T04:11:57.0310059Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp97sksb7l/_remote_module_non_scriptable.py
2022-05-18T04:11:57.0582687Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyw8jk4my
2022-05-18T04:11:57.0583600Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyw8jk4my/_remote_module_non_scriptable.py
2022-05-18T04:11:57.0865179Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvljbqget
2022-05-18T04:11:57.0866540Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvljbqget/_remote_module_non_scriptable.py
2022-05-18T04:11:57.1006869Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2suadh25
2022-05-18T04:11:57.1007655Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2suadh25/_remote_module_non_scriptable.py
2022-05-18T04:11:57.2882239Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:11:57.3133197Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:11:57.3459006Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:11:57.3597859Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:11:57.9436142Z ok (1.845s)
2022-05-18T04:11:57.9436372Z 
2022-05-18T04:11:57.9436764Z ----------------------------------------------------------------------
2022-05-18T04:11:57.9437022Z Ran 1 test in 1.845s
2022-05-18T04:11:57.9437139Z 
2022-05-18T04:11:57.9437202Z OK
2022-05-18T04:11:57.9437295Z 
2022-05-18T04:11:57.9437392Z Generating XML reports...
2022-05-18T04:11:57.9472653Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220518041156.xml
2022-05-18T04:11:58.7557046Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpecdaf7n2
2022-05-18T04:11:58.7557529Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpecdaf7n2/_remote_module_non_scriptable.py
2022-05-18T04:11:59.0123992Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:11:59.0133284Z 
2022-05-18T04:11:59.0133403Z Running tests...
2022-05-18T04:11:59.0134019Z ----------------------------------------------------------------------
2022-05-18T04:11:59.3459310Z   test_py_rref_args_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31336
2022-05-18T04:11:59.3482628Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31337
2022-05-18T04:11:59.3506665Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31338
2022-05-18T04:11:59.3531705Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 31339
2022-05-18T04:11:59.9515317Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp69epgink
2022-05-18T04:11:59.9516222Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp69epgink/_remote_module_non_scriptable.py
2022-05-18T04:11:59.9986541Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptfapjklp
2022-05-18T04:11:59.9987332Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptfapjklp/_remote_module_non_scriptable.py
2022-05-18T04:12:00.0139108Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplvepfn6x
2022-05-18T04:12:00.0139929Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplvepfn6x/_remote_module_non_scriptable.py
2022-05-18T04:12:00.0299638Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcf5rt5ex
2022-05-18T04:12:00.0300427Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcf5rt5ex/_remote_module_non_scriptable.py
2022-05-18T04:12:00.2067397Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:12:00.2498591Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:12:00.2675268Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:12:00.2820945Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:12:00.7572400Z ok (1.744s)
2022-05-18T04:12:00.7572650Z 
2022-05-18T04:12:00.7573165Z ----------------------------------------------------------------------
2022-05-18T04:12:00.7573593Z Ran 1 test in 1.744s
2022-05-18T04:12:00.7573711Z 
2022-05-18T04:12:00.7573772Z OK
2022-05-18T04:12:00.7573862Z 
2022-05-18T04:12:00.7573982Z Generating XML reports...
2022-05-18T04:12:00.7608991Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220518041159.xml
2022-05-18T04:12:01.5890116Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3kw2j5hg
2022-05-18T04:12:01.5891088Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3kw2j5hg/_remote_module_non_scriptable.py
2022-05-18T04:12:01.8519305Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:12:01.8529131Z 
2022-05-18T04:12:01.8529221Z Running tests...
2022-05-18T04:12:01.8530142Z ----------------------------------------------------------------------
2022-05-18T04:12:02.1873132Z   test_py_rref_args_user_share_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31567
2022-05-18T04:12:02.1897274Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31568
2022-05-18T04:12:02.1920476Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31569
2022-05-18T04:12:02.1945126Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 31570
2022-05-18T04:12:02.8588097Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm7yrs_1w
2022-05-18T04:12:02.8588876Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm7yrs_1w/_remote_module_non_scriptable.py
2022-05-18T04:12:02.8833972Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpixclnw76
2022-05-18T04:12:02.8834699Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpixclnw76/_remote_module_non_scriptable.py
2022-05-18T04:12:02.9196825Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf2g49mg6
2022-05-18T04:12:02.9197551Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf2g49mg6/_remote_module_non_scriptable.py
2022-05-18T04:12:02.9295028Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe3toe645
2022-05-18T04:12:02.9295964Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe3toe645/_remote_module_non_scriptable.py
2022-05-18T04:12:03.1148637Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:12:03.1397630Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:12:03.1763179Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:12:03.1832673Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:12:03.7990626Z ok (1.946s)
2022-05-18T04:12:03.7990884Z 
2022-05-18T04:12:03.7991394Z ----------------------------------------------------------------------
2022-05-18T04:12:03.7991857Z Ran 1 test in 1.946s
2022-05-18T04:12:03.7991990Z 
2022-05-18T04:12:03.7992052Z OK
2022-05-18T04:12:03.7992159Z 
2022-05-18T04:12:03.7992253Z Generating XML reports...
2022-05-18T04:12:03.8028835Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220518041201.xml
2022-05-18T04:12:04.6285134Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9mqj9k5p
2022-05-18T04:12:04.6286046Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9mqj9k5p/_remote_module_non_scriptable.py
2022-05-18T04:12:04.8896478Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:12:04.8906321Z 
2022-05-18T04:12:04.8906453Z Running tests...
2022-05-18T04:12:04.8907024Z ----------------------------------------------------------------------
2022-05-18T04:12:05.2244158Z   test_py_sparse_tensors_in_container (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31798
2022-05-18T04:12:05.2268309Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31799
2022-05-18T04:12:05.2292664Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31800
2022-05-18T04:12:05.2317992Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 31801
2022-05-18T04:12:05.8457555Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3bw60yi3
2022-05-18T04:12:05.8458720Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3bw60yi3/_remote_module_non_scriptable.py
2022-05-18T04:12:05.8476910Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt6bdcsps
2022-05-18T04:12:05.8478958Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt6bdcsps/_remote_module_non_scriptable.py
2022-05-18T04:12:05.8669414Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgfw65aey
2022-05-18T04:12:05.8670838Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgfw65aey/_remote_module_non_scriptable.py
2022-05-18T04:12:05.8988162Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq3ofxb_4
2022-05-18T04:12:05.8988942Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq3ofxb_4/_remote_module_non_scriptable.py
2022-05-18T04:12:06.0986679Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:12:06.1026090Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:12:06.1196022Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:12:06.1506885Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:12:06.7361433Z ok (1.845s)
2022-05-18T04:12:06.7361701Z 
2022-05-18T04:12:06.7362115Z ----------------------------------------------------------------------
2022-05-18T04:12:06.7362369Z Ran 1 test in 1.845s
2022-05-18T04:12:06.7362487Z 
2022-05-18T04:12:06.7362825Z OK
2022-05-18T04:12:06.7362924Z 
2022-05-18T04:12:06.7363021Z Generating XML reports...
2022-05-18T04:12:06.7397150Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220518041204.xml
2022-05-18T04:12:07.5708771Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp811vju7
2022-05-18T04:12:07.5709225Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp811vju7/_remote_module_non_scriptable.py
2022-05-18T04:12:07.8306773Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:12:07.8316878Z 
2022-05-18T04:12:07.8317329Z Running tests...
2022-05-18T04:12:07.8317762Z ----------------------------------------------------------------------
2022-05-18T04:12:08.1655039Z   test_rref_get_type_timeout_blocking (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32029
2022-05-18T04:12:08.1678634Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32030
2022-05-18T04:12:08.1702614Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 32031
2022-05-18T04:12:08.1727220Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 32032
2022-05-18T04:12:08.8041849Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpviy2c_76
2022-05-18T04:12:08.8042980Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpphpvh_cv
2022-05-18T04:12:08.8044035Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpviy2c_76/_remote_module_non_scriptable.py
2022-05-18T04:12:08.8044929Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpphpvh_cv/_remote_module_non_scriptable.py
2022-05-18T04:12:08.8420972Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_1ob52cl
2022-05-18T04:12:08.8421775Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_1ob52cl/_remote_module_non_scriptable.py
2022-05-18T04:12:08.8550460Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9wb28shl
2022-05-18T04:12:08.8551825Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9wb28shl/_remote_module_non_scriptable.py
2022-05-18T04:12:09.0570860Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:12:09.0581794Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:12:09.0937405Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:12:09.1066236Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:12:11.5799527Z ok (3.748s)
2022-05-18T04:12:11.5799806Z 
2022-05-18T04:12:11.5800289Z ----------------------------------------------------------------------
2022-05-18T04:12:11.5800547Z Ran 1 test in 3.748s
2022-05-18T04:12:11.5800677Z 
2022-05-18T04:12:11.5800725Z OK
2022-05-18T04:12:11.5800822Z 
2022-05-18T04:12:11.5800917Z Generating XML reports...
2022-05-18T04:12:11.5834335Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220518041207.xml
2022-05-18T04:12:12.3829302Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8ihup2bl
2022-05-18T04:12:12.3829782Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8ihup2bl/_remote_module_non_scriptable.py
2022-05-18T04:12:12.6393785Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:12:12.6403298Z 
2022-05-18T04:12:12.6403653Z Running tests...
2022-05-18T04:12:12.6404292Z ----------------------------------------------------------------------
2022-05-18T04:12:12.9666309Z   test_rref_get_type_timeout_non_blocking (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32248
2022-05-18T04:12:12.9690673Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32249
2022-05-18T04:12:12.9715869Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 32250
2022-05-18T04:12:12.9741082Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 32251
2022-05-18T04:12:13.6459764Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpftofgdth
2022-05-18T04:12:13.6460579Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpftofgdth/_remote_module_non_scriptable.py
2022-05-18T04:12:13.6474589Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmponw5fu7y
2022-05-18T04:12:13.6476113Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmponw5fu7y/_remote_module_non_scriptable.py
2022-05-18T04:12:13.6602514Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9ji3mawk
2022-05-18T04:12:13.6604588Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9ji3mawk/_remote_module_non_scriptable.py
2022-05-18T04:12:13.6962261Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgc4k5oc0
2022-05-18T04:12:13.6963440Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgc4k5oc0/_remote_module_non_scriptable.py
2022-05-18T04:12:13.8983758Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:12:13.8994427Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:12:13.9116308Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:12:13.9489354Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:12:16.4814926Z ok (3.841s)
2022-05-18T04:12:16.4815210Z 
2022-05-18T04:12:16.4815726Z ----------------------------------------------------------------------
2022-05-18T04:12:16.4816006Z Ran 1 test in 3.841s
2022-05-18T04:12:16.4816109Z 
2022-05-18T04:12:16.4816171Z OK
2022-05-18T04:12:16.4816266Z 
2022-05-18T04:12:16.4816361Z Generating XML reports...
2022-05-18T04:12:16.4850616Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220518041212.xml
2022-05-18T04:12:17.2903178Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkpitf_5t
2022-05-18T04:12:17.2904175Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkpitf_5t/_remote_module_non_scriptable.py
2022-05-18T04:12:17.5511488Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:12:17.5521224Z 
2022-05-18T04:12:17.5521347Z Running tests...
2022-05-18T04:12:17.5522522Z ----------------------------------------------------------------------
2022-05-18T04:12:17.8802182Z   test_rref_proxy_timeout (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32467
2022-05-18T04:12:17.8825927Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32468
2022-05-18T04:12:17.8849509Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 32469
2022-05-18T04:12:17.8874104Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 32470
2022-05-18T04:12:18.5610725Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpby82sp7h
2022-05-18T04:12:18.5611488Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpby82sp7h/_remote_module_non_scriptable.py
2022-05-18T04:12:18.5910981Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7tnjv4b7
2022-05-18T04:12:18.5911719Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7tnjv4b7/_remote_module_non_scriptable.py
2022-05-18T04:12:18.6301333Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0qxewd8o
2022-05-18T04:12:18.6302317Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0qxewd8o/_remote_module_non_scriptable.py
2022-05-18T04:12:18.6673155Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvtrn41sx
2022-05-18T04:12:18.6673989Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvtrn41sx/_remote_module_non_scriptable.py
2022-05-18T04:12:18.8155547Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:12:18.8453348Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:12:18.8825825Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:12:18.9202166Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:12:31.1813683Z [W tensorpipe_agent.cpp:942] RPC agent for worker3 encountered error when reading incoming response from worker0: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2022-05-18T04:12:32.1284723Z [W tensorpipe_agent.cpp:627] RPC agent for worker1 won't send response to request #30 to worker0, as the agent is shutting down
2022-05-18T04:12:32.1509994Z [W tensorpipe_agent.cpp:627] RPC agent for worker0 won't send response to request #28 to worker3, as the agent is shutting down
2022-05-18T04:12:32.1539092Z [W tensorpipe_agent.cpp:627] RPC agent for worker3 won't send response to request #28 to worker2, as the agent is shutting down
2022-05-18T04:12:32.1587612Z [W tensorpipe_agent.cpp:627] RPC agent for worker2 won't send response to request #30 to worker1, as the agent is shutting down
2022-05-18T04:12:32.4125055Z ok (14.860s)
2022-05-18T04:12:32.4125330Z 
2022-05-18T04:12:32.4125810Z ----------------------------------------------------------------------
2022-05-18T04:12:32.4126062Z Ran 1 test in 14.860s
2022-05-18T04:12:32.4126165Z 
2022-05-18T04:12:32.4126228Z OK
2022-05-18T04:12:32.4126322Z 
2022-05-18T04:12:32.4126432Z Generating XML reports...
2022-05-18T04:12:32.4160814Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220518041217.xml
2022-05-18T04:12:33.2365039Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp26irm2sv
2022-05-18T04:12:33.2365821Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp26irm2sv/_remote_module_non_scriptable.py
2022-05-18T04:12:33.4981562Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:12:33.4991627Z 
2022-05-18T04:12:33.4991881Z Running tests...
2022-05-18T04:12:33.4992497Z ----------------------------------------------------------------------
2022-05-18T04:12:33.8339584Z   test_self_py_udf_remote_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32686
2022-05-18T04:12:33.8365041Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32687
2022-05-18T04:12:33.8388664Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 32688
2022-05-18T04:12:33.8413964Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 32689
2022-05-18T04:12:34.4265692Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0skyqqs3
2022-05-18T04:12:34.4267644Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0skyqqs3/_remote_module_non_scriptable.py
2022-05-18T04:12:34.4644079Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2eddwjmd
2022-05-18T04:12:34.4644870Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2eddwjmd/_remote_module_non_scriptable.py
2022-05-18T04:12:34.4984329Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3j86c6gq
2022-05-18T04:12:34.4985430Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3j86c6gq/_remote_module_non_scriptable.py
2022-05-18T04:12:34.5006087Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpka3232fw
2022-05-18T04:12:34.5008204Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpka3232fw/_remote_module_non_scriptable.py
2022-05-18T04:12:34.6833081Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:12:34.7183809Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:12:34.7529669Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:12:34.7562953Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:12:35.3457656Z ok (1.846s)
2022-05-18T04:12:35.3457915Z 
2022-05-18T04:12:35.3458414Z ----------------------------------------------------------------------
2022-05-18T04:12:35.3458829Z Ran 1 test in 1.847s
2022-05-18T04:12:35.3458944Z 
2022-05-18T04:12:35.3459005Z OK
2022-05-18T04:12:35.3459114Z 
2022-05-18T04:12:35.3459210Z Generating XML reports...
2022-05-18T04:12:35.3493414Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220518041233.xml
2022-05-18T04:12:36.1721584Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph6_s3n4a
2022-05-18T04:12:36.1722331Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph6_s3n4a/_remote_module_non_scriptable.py
2022-05-18T04:12:36.4331688Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:12:36.4342067Z 
2022-05-18T04:12:36.4342426Z Running tests...
2022-05-18T04:12:36.4343333Z ----------------------------------------------------------------------
2022-05-18T04:12:36.7684592Z   test_self_remote_rref_as_remote_arg_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 449
2022-05-18T04:12:36.7709037Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 450
2022-05-18T04:12:36.7733482Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 451
2022-05-18T04:12:36.7760197Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 452
2022-05-18T04:12:37.3707017Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6bv_1ckx
2022-05-18T04:12:37.3707779Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6bv_1ckx/_remote_module_non_scriptable.py
2022-05-18T04:12:37.3832054Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph15ca5vd
2022-05-18T04:12:37.3832855Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph15ca5vd/_remote_module_non_scriptable.py
2022-05-18T04:12:37.4389126Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpts9u3h0a
2022-05-18T04:12:37.4389956Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1n18jaaf
2022-05-18T04:12:37.4390597Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpts9u3h0a/_remote_module_non_scriptable.py
2022-05-18T04:12:37.4391220Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1n18jaaf/_remote_module_non_scriptable.py
2022-05-18T04:12:37.6218239Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:12:37.6359756Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:12:37.6913042Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:12:37.6920068Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:12:38.1799363Z ok (1.745s)
2022-05-18T04:12:38.1799595Z 
2022-05-18T04:12:38.1799917Z ----------------------------------------------------------------------
2022-05-18T04:12:38.1800161Z Ran 1 test in 1.746s
2022-05-18T04:12:38.1800325Z 
2022-05-18T04:12:38.1800661Z OK
2022-05-18T04:12:38.1800755Z 
2022-05-18T04:12:38.1800851Z Generating XML reports...
2022-05-18T04:12:38.1834681Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220518041236.xml
2022-05-18T04:12:39.0074284Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvmlw393x
2022-05-18T04:12:39.0075268Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvmlw393x/_remote_module_non_scriptable.py
2022-05-18T04:12:39.2684754Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:12:39.2694685Z 
2022-05-18T04:12:39.2694820Z Running tests...
2022-05-18T04:12:39.2695277Z ----------------------------------------------------------------------
2022-05-18T04:12:39.6046064Z   test_self_remote_rref_as_rpc_arg_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 680
2022-05-18T04:12:39.6069712Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 681
2022-05-18T04:12:39.6093223Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 682
2022-05-18T04:12:39.6118449Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 683
2022-05-18T04:12:40.2848177Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptgo7txjo
2022-05-18T04:12:40.2848919Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptgo7txjo/_remote_module_non_scriptable.py
2022-05-18T04:12:40.2985061Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpur1ishjy
2022-05-18T04:12:40.2985805Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpur1ishjy/_remote_module_non_scriptable.py
2022-05-18T04:12:40.3303097Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8jtug_bd
2022-05-18T04:12:40.3303870Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8jtug_bd/_remote_module_non_scriptable.py
2022-05-18T04:12:40.3328101Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6bejv0nw
2022-05-18T04:12:40.3329884Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6bejv0nw/_remote_module_non_scriptable.py
2022-05-18T04:12:40.5386489Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:12:40.5520259Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:12:40.5824060Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:12:40.5860165Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:12:41.1161924Z ok (1.846s)
2022-05-18T04:12:41.1162101Z 
2022-05-18T04:12:41.1162467Z ----------------------------------------------------------------------
2022-05-18T04:12:41.1162718Z Ran 1 test in 1.847s
2022-05-18T04:12:41.1162860Z 
2022-05-18T04:12:41.1162922Z OK
2022-05-18T04:12:41.1163018Z 
2022-05-18T04:12:41.1163114Z Generating XML reports...
2022-05-18T04:12:41.1197152Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220518041239.xml
2022-05-18T04:12:41.9224859Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_rtxo22_
2022-05-18T04:12:41.9225958Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_rtxo22_/_remote_module_non_scriptable.py
2022-05-18T04:12:42.1803152Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:12:42.1813270Z 
2022-05-18T04:12:42.1813516Z Running tests...
2022-05-18T04:12:42.1814141Z ----------------------------------------------------------------------
2022-05-18T04:12:42.5101961Z   test_self_remote_rref_as_self_remote_arg_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 911
2022-05-18T04:12:42.5126116Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 912
2022-05-18T04:12:42.5150438Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 913
2022-05-18T04:12:42.5175558Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 914
2022-05-18T04:12:43.1115715Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt27qdgge
2022-05-18T04:12:43.1116531Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt27qdgge/_remote_module_non_scriptable.py
2022-05-18T04:12:43.1317762Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpisto5cj5
2022-05-18T04:12:43.1318513Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpisto5cj5/_remote_module_non_scriptable.py
2022-05-18T04:12:43.1570649Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplnn0aai9
2022-05-18T04:12:43.1571559Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplnn0aai9/_remote_module_non_scriptable.py
2022-05-18T04:12:43.1760905Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn2l9dl06
2022-05-18T04:12:43.1761689Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn2l9dl06/_remote_module_non_scriptable.py
2022-05-18T04:12:43.3660063Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:12:43.3867542Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:12:43.4087904Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:12:43.4305168Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:12:43.9216077Z ok (1.740s)
2022-05-18T04:12:43.9216344Z 
2022-05-18T04:12:43.9216880Z ----------------------------------------------------------------------
2022-05-18T04:12:43.9217262Z Ran 1 test in 1.740s
2022-05-18T04:12:43.9217379Z 
2022-05-18T04:12:43.9217428Z OK
2022-05-18T04:12:43.9217521Z 
2022-05-18T04:12:43.9217618Z Generating XML reports...
2022-05-18T04:12:43.9251773Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220518041242.xml
2022-05-18T04:12:44.7309950Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8h6czb11
2022-05-18T04:12:44.7310950Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8h6czb11/_remote_module_non_scriptable.py
2022-05-18T04:12:44.9894317Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:12:44.9904128Z 
2022-05-18T04:12:44.9904281Z Running tests...
2022-05-18T04:12:44.9904680Z ----------------------------------------------------------------------
2022-05-18T04:12:45.3194987Z   test_self_remote_rref_as_self_rpc_arg_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1142
2022-05-18T04:12:45.3218506Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1143
2022-05-18T04:12:45.3241818Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1144
2022-05-18T04:12:45.3267196Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1145
2022-05-18T04:12:46.0185595Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpczz_vvj_
2022-05-18T04:12:46.0186352Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpczz_vvj_/_remote_module_non_scriptable.py
2022-05-18T04:12:46.0317809Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3tebxdt3
2022-05-18T04:12:46.0318824Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3tebxdt3/_remote_module_non_scriptable.py
2022-05-18T04:12:46.0419095Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq35uocs1
2022-05-18T04:12:46.0420118Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq35uocs1/_remote_module_non_scriptable.py
2022-05-18T04:12:46.0624144Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmv_vv1wp
2022-05-18T04:12:46.0624896Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmv_vv1wp/_remote_module_non_scriptable.py
2022-05-18T04:12:46.2696122Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:12:46.2851693Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:12:46.2931748Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:12:46.3159009Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:12:46.8310109Z ok (1.840s)
2022-05-18T04:12:46.8310365Z 
2022-05-18T04:12:46.8310830Z ----------------------------------------------------------------------
2022-05-18T04:12:46.8311093Z Ran 1 test in 1.841s
2022-05-18T04:12:46.8311211Z 
2022-05-18T04:12:46.8311260Z OK
2022-05-18T04:12:46.8311352Z 
2022-05-18T04:12:46.8311444Z Generating XML reports...
2022-05-18T04:12:46.8345642Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220518041244.xml
2022-05-18T04:12:47.6348305Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj8eidyeh
2022-05-18T04:12:47.6349091Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj8eidyeh/_remote_module_non_scriptable.py
2022-05-18T04:12:47.8907556Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:12:47.8917385Z 
2022-05-18T04:12:47.8917516Z Running tests...
2022-05-18T04:12:47.8917937Z ----------------------------------------------------------------------
2022-05-18T04:12:48.2207175Z   test_send_to_rank_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1373
2022-05-18T04:12:48.2229912Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1374
2022-05-18T04:12:48.2253643Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1375
2022-05-18T04:12:48.2278393Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1376
2022-05-18T04:12:48.8585050Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnzkl0_sz
2022-05-18T04:12:48.8585877Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnzkl0_sz/_remote_module_non_scriptable.py
2022-05-18T04:12:48.8679541Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpiaglkh_g
2022-05-18T04:12:48.8680893Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpiaglkh_g/_remote_module_non_scriptable.py
2022-05-18T04:12:48.8694640Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpetszlbv8
2022-05-18T04:12:48.8696271Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpetszlbv8/_remote_module_non_scriptable.py
2022-05-18T04:12:48.8802118Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpslo2mf16
2022-05-18T04:12:48.8803086Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpslo2mf16/_remote_module_non_scriptable.py
2022-05-18T04:12:49.1116080Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:12:49.1202939Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:12:49.1203587Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:12:49.1330040Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:12:49.7321202Z ok (1.840s)
2022-05-18T04:12:49.7321458Z 
2022-05-18T04:12:49.7321915Z ----------------------------------------------------------------------
2022-05-18T04:12:49.7322634Z Ran 1 test in 1.840s
2022-05-18T04:12:49.7322811Z 
2022-05-18T04:12:49.7322910Z OK
2022-05-18T04:12:49.7323054Z 
2022-05-18T04:12:49.7323333Z Generating XML reports...
2022-05-18T04:12:49.7358378Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220518041247.xml
2022-05-18T04:12:50.5427316Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1nqcm9ok
2022-05-18T04:12:50.5427880Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1nqcm9ok/_remote_module_non_scriptable.py
2022-05-18T04:12:50.7993560Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:12:50.8003289Z 
2022-05-18T04:12:50.8003762Z Running tests...
2022-05-18T04:12:50.8004166Z ----------------------------------------------------------------------
2022-05-18T04:12:51.1351154Z   test_set_and_get_num_worker_threads (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1604
2022-05-18T04:12:51.1376877Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1605
2022-05-18T04:12:51.1401038Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1606
2022-05-18T04:12:51.1425806Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1607
2022-05-18T04:12:51.8211040Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb55zkqs1
2022-05-18T04:12:51.8211759Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb55zkqs1/_remote_module_non_scriptable.py
2022-05-18T04:12:51.8748214Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpca2epzyg
2022-05-18T04:12:51.8749644Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpca2epzyg/_remote_module_non_scriptable.py
2022-05-18T04:12:51.8792278Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6qprd6xv
2022-05-18T04:12:51.8793779Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6qprd6xv/_remote_module_non_scriptable.py
2022-05-18T04:12:51.8923191Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe2j899d_
2022-05-18T04:12:51.8924430Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe2j899d_/_remote_module_non_scriptable.py
2022-05-18T04:12:52.0741423Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:12:52.1242418Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:12:52.1297495Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:12:52.1399691Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:12:52.6468038Z ok (1.846s)
2022-05-18T04:12:52.6468253Z 
2022-05-18T04:12:52.6468794Z ----------------------------------------------------------------------
2022-05-18T04:12:52.6469211Z Ran 1 test in 1.846s
2022-05-18T04:12:52.6469343Z 
2022-05-18T04:12:52.6469405Z OK
2022-05-18T04:12:52.6469495Z 
2022-05-18T04:12:52.6469595Z Generating XML reports...
2022-05-18T04:12:52.6503101Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220518041250.xml
2022-05-18T04:12:53.4301780Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpaghbsy0k
2022-05-18T04:12:53.4302464Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpaghbsy0k/_remote_module_non_scriptable.py
2022-05-18T04:12:53.6841428Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:12:53.6850679Z 
2022-05-18T04:12:53.6850798Z Running tests...
2022-05-18T04:12:53.6851466Z ----------------------------------------------------------------------
2022-05-18T04:12:54.0067644Z   test_stress_heavy_rpc_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1867
2022-05-18T04:12:54.0091338Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1868
2022-05-18T04:12:54.0113968Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1869
2022-05-18T04:12:54.0137732Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1870
2022-05-18T04:12:54.6782316Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjjzssp8p
2022-05-18T04:12:54.6783244Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjjzssp8p/_remote_module_non_scriptable.py
2022-05-18T04:12:54.7058794Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc39373qh
2022-05-18T04:12:54.7059578Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc39373qh/_remote_module_non_scriptable.py
2022-05-18T04:12:54.7440008Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr3wcz0nm
2022-05-18T04:12:54.7440819Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr3wcz0nm/_remote_module_non_scriptable.py
2022-05-18T04:12:54.7471832Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppo2wi0sf
2022-05-18T04:12:54.7473455Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppo2wi0sf/_remote_module_non_scriptable.py
2022-05-18T04:12:54.9299661Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:12:54.9534794Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:12:54.9924631Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:12:54.9966202Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:12:55.3124100Z Rank 0 finished testing 20 times in 0.10379147529602051 seconds.
2022-05-18T04:12:55.3354852Z Rank 3 finished testing 20 times in 0.09114313125610352 seconds.
2022-05-18T04:12:55.3404958Z Rank 1 finished testing 20 times in 0.08900594711303711 seconds.
2022-05-18T04:12:55.3501582Z Rank 2 finished testing 20 times in 0.09123396873474121 seconds.
2022-05-18T04:12:55.6181185Z ok (1.933s)
2022-05-18T04:12:55.6181380Z 
2022-05-18T04:12:55.6181776Z ----------------------------------------------------------------------
2022-05-18T04:12:55.6182075Z Ran 1 test in 1.933s
2022-05-18T04:12:55.6182193Z 
2022-05-18T04:12:55.6182259Z OK
2022-05-18T04:12:55.6182352Z 
2022-05-18T04:12:55.6182446Z Generating XML reports...
2022-05-18T04:12:55.6216414Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220518041253.xml
2022-05-18T04:12:56.4060339Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1128dnpl
2022-05-18T04:12:56.4061103Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1128dnpl/_remote_module_non_scriptable.py
2022-05-18T04:12:56.6598961Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:12:56.6608936Z 
2022-05-18T04:12:56.6609339Z Running tests...
2022-05-18T04:12:56.6609734Z ----------------------------------------------------------------------
2022-05-18T04:12:56.9791742Z   test_tensorpipe_options_throw_on_timedelta_timeout (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2098
2022-05-18T04:12:56.9815337Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2099
2022-05-18T04:12:56.9838391Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2100
2022-05-18T04:12:56.9863026Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2101
2022-05-18T04:12:57.6895776Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4v4cuzeg
2022-05-18T04:12:57.6896725Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4v4cuzeg/_remote_module_non_scriptable.py
2022-05-18T04:12:57.6967277Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6jaw73z3
2022-05-18T04:12:57.6968051Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6jaw73z3/_remote_module_non_scriptable.py
2022-05-18T04:12:57.6979260Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpettng8m4
2022-05-18T04:12:57.6980802Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpettng8m4/_remote_module_non_scriptable.py
2022-05-18T04:12:57.7018863Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6pl5pa3r
2022-05-18T04:12:57.7021487Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6pl5pa3r/_remote_module_non_scriptable.py
2022-05-18T04:12:57.9394016Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:12:57.9459722Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:12:57.9499754Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:12:57.9526705Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:12:58.1899170Z ok (1.529s)
2022-05-18T04:12:58.1899436Z 
2022-05-18T04:12:58.1899953Z ----------------------------------------------------------------------
2022-05-18T04:12:58.1900217Z Ran 1 test in 1.529s
2022-05-18T04:12:58.1900330Z 
2022-05-18T04:12:58.1900393Z OK
2022-05-18T04:12:58.1900488Z 
2022-05-18T04:12:58.1900582Z Generating XML reports...
2022-05-18T04:12:58.1934243Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220518041256.xml
2022-05-18T04:12:58.9743830Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_4tzsvi1
2022-05-18T04:12:58.9744755Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_4tzsvi1/_remote_module_non_scriptable.py
2022-05-18T04:12:59.2301694Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:12:59.2311740Z 
2022-05-18T04:12:59.2311824Z Running tests...
2022-05-18T04:12:59.2312245Z ----------------------------------------------------------------------
2022-05-18T04:12:59.5587655Z   test_tensorpipe_set_default_timeout (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2153
2022-05-18T04:12:59.5610509Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2154
2022-05-18T04:12:59.5633940Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2155
2022-05-18T04:12:59.5658791Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2156
2022-05-18T04:13:00.1982685Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpptsnsu7f
2022-05-18T04:13:00.2029247Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpptsnsu7f/_remote_module_non_scriptable.py
2022-05-18T04:13:00.2094072Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5a4uos5r
2022-05-18T04:13:00.2094720Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpviiigo5k
2022-05-18T04:13:00.2095333Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5a4uos5r/_remote_module_non_scriptable.py
2022-05-18T04:13:00.2095750Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpviiigo5k/_remote_module_non_scriptable.py
2022-05-18T04:13:00.2598175Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphjlkdl7i
2022-05-18T04:13:00.2599035Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphjlkdl7i/_remote_module_non_scriptable.py
2022-05-18T04:13:00.4587105Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:13:00.4624998Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:13:00.4980250Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:13:00.5358011Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:13:00.9699061Z ok (1.738s)
2022-05-18T04:13:00.9699291Z 
2022-05-18T04:13:00.9699722Z ----------------------------------------------------------------------
2022-05-18T04:13:00.9700128Z Ran 1 test in 1.739s
2022-05-18T04:13:00.9700322Z 
2022-05-18T04:13:00.9700440Z OK
2022-05-18T04:13:00.9700610Z 
2022-05-18T04:13:00.9700728Z Generating XML reports...
2022-05-18T04:13:00.9736977Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220518041259.xml
2022-05-18T04:13:01.7787443Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2am5ugrt
2022-05-18T04:13:01.7788460Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2am5ugrt/_remote_module_non_scriptable.py
2022-05-18T04:13:02.0370371Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:13:02.0379959Z 
2022-05-18T04:13:02.0380088Z Running tests...
2022-05-18T04:13:02.0380695Z ----------------------------------------------------------------------
2022-05-18T04:13:02.3688693Z   test_wait_all_workers_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2372
2022-05-18T04:13:02.3711839Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2373
2022-05-18T04:13:02.3735280Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2374
2022-05-18T04:13:02.3759816Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2375
2022-05-18T04:13:02.9967621Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppsa3o73_
2022-05-18T04:13:02.9968768Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppsa3o73_/_remote_module_non_scriptable.py
2022-05-18T04:13:03.0287295Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpry6wcqrh
2022-05-18T04:13:03.0288032Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpry6wcqrh/_remote_module_non_scriptable.py
2022-05-18T04:13:03.0332825Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6bcumxb6
2022-05-18T04:13:03.0334290Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6bcumxb6/_remote_module_non_scriptable.py
2022-05-18T04:13:03.0349397Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnjzwdu1o
2022-05-18T04:13:03.0351635Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnjzwdu1o/_remote_module_non_scriptable.py
2022-05-18T04:13:03.2492000Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:13:03.2831890Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:13:03.2867509Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:13:03.2882488Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:13:03.3087851Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:13:03.3188466Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:13:03.3290059Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T04:13:03.3290603Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:13:03.3291790Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:13:03.3292868Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:13:03.3293782Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:13:03.3294637Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:13:03.7164215Z [W tensorpipe_agent.cpp:728] RPC agent for worker2 encountered error when reading incoming request from worker0: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2022-05-18T04:13:03.7165068Z [W tensorpipe_agent.cpp:728] RPC agent for worker3 encountered error when reading incoming request from worker0: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2022-05-18T04:13:03.7173486Z [W tensorpipe_agent.cpp:728] RPC agent for worker1 encountered error when reading incoming request from worker0: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2022-05-18T04:13:03.7231090Z [W tensorpipe_agent.cpp:728] RPC agent for worker0 encountered error when reading incoming request from worker2: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2022-05-18T04:13:03.7231658Z [W tensorpipe_agent.cpp:728] RPC agent for worker0 encountered error when reading incoming request from worker1: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2022-05-18T04:13:03.7232189Z [W tensorpipe_agent.cpp:728] RPC agent for worker0 encountered error when reading incoming request from worker3: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2022-05-18T04:13:03.9803127Z ok (1.942s)
2022-05-18T04:13:03.9803471Z 
2022-05-18T04:13:03.9803987Z ----------------------------------------------------------------------
2022-05-18T04:13:03.9804241Z Ran 1 test in 1.942s
2022-05-18T04:13:03.9804383Z 
2022-05-18T04:13:03.9804445Z OK
2022-05-18T04:13:03.9804524Z 
2022-05-18T04:13:03.9804618Z Generating XML reports...
2022-05-18T04:13:03.9840658Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220518041302.xml
2022-05-18T04:13:04.7817370Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvd7_lubd
2022-05-18T04:13:04.7817985Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvd7_lubd/_remote_module_non_scriptable.py
2022-05-18T04:13:05.0354311Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:13:05.0364123Z 
2022-05-18T04:13:05.0364240Z Running tests...
2022-05-18T04:13:05.0364636Z ----------------------------------------------------------------------
2022-05-18T04:13:05.3581124Z   test_wait_all_workers_twice_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2615
2022-05-18T04:13:05.3604523Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2616
2022-05-18T04:13:05.3627951Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2617
2022-05-18T04:13:05.3652756Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2618
2022-05-18T04:13:05.9918780Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4n2kdzer
2022-05-18T04:13:05.9919815Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4n2kdzer/_remote_module_non_scriptable.py
2022-05-18T04:13:05.9996035Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt6ddvvbd
2022-05-18T04:13:05.9997862Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt6ddvvbd/_remote_module_non_scriptable.py
2022-05-18T04:13:06.0057358Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppw2zutf5
2022-05-18T04:13:06.0059297Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppw2zutf5/_remote_module_non_scriptable.py
2022-05-18T04:13:06.0112991Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy1kgzwpr
2022-05-18T04:13:06.0114141Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy1kgzwpr/_remote_module_non_scriptable.py
2022-05-18T04:13:06.2431805Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:13:06.2522805Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:13:06.2561715Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:13:06.2608153Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:13:06.2940630Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:13:06.2998052Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T04:13:06.2998788Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:13:06.2999598Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:13:06.3000007Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:13:06.3000490Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:13:06.3001019Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:13:06.3041430Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:13:06.6912978Z [W tensorpipe_agent.cpp:728] RPC agent for worker2 encountered error when reading incoming request from worker0: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2022-05-18T04:13:06.6916653Z [W tensorpipe_agent.cpp:728] RPC agent for worker0 encountered error when reading incoming request from worker2: eof (this error originated at tensorpipe/transport/shm/connection_impl.cc:259)
2022-05-18T04:13:06.6917641Z [W tensorpipe_agent.cpp:728] RPC agent for worker1 encountered error when reading incoming request from worker0: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2022-05-18T04:13:06.6920249Z [W tensorpipe_agent.cpp:728] RPC agent for worker3 encountered error when reading incoming request from worker0: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2022-05-18T04:13:06.6946793Z [W tensorpipe_agent.cpp:728] RPC agent for worker0 encountered error when reading incoming request from worker3: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2022-05-18T04:13:06.6947765Z [W tensorpipe_agent.cpp:728] RPC agent for worker0 encountered error when reading incoming request from worker1: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2022-05-18T04:13:06.9696493Z ok (1.933s)
2022-05-18T04:13:06.9696696Z 
2022-05-18T04:13:06.9697172Z ----------------------------------------------------------------------
2022-05-18T04:13:06.9697606Z Ran 1 test in 1.933s
2022-05-18T04:13:06.9697816Z 
2022-05-18T04:13:06.9697915Z OK
2022-05-18T04:13:06.9698076Z 
2022-05-18T04:13:06.9698199Z Generating XML reports...
2022-05-18T04:13:06.9732277Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220518041305.xml
2022-05-18T04:13:07.7594008Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqsv8jwpl
2022-05-18T04:13:07.7595156Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqsv8jwpl/_remote_module_non_scriptable.py
2022-05-18T04:13:08.0159827Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:13:08.0169466Z 
2022-05-18T04:13:08.0169564Z Running tests...
2022-05-18T04:13:08.0170050Z ----------------------------------------------------------------------
2022-05-18T04:13:08.3430219Z   test_world_size_one_sparse (__main__.TensorPipeTensorPipeAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2858
2022-05-18T04:13:08.3452477Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2859
2022-05-18T04:13:08.3476700Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2860
2022-05-18T04:13:08.3501260Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2861
2022-05-18T04:13:08.9480435Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy7l9xx41
2022-05-18T04:13:08.9481218Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy7l9xx41/_remote_module_non_scriptable.py
2022-05-18T04:13:08.9738025Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9u9tjwvj
2022-05-18T04:13:08.9738823Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9u9tjwvj/_remote_module_non_scriptable.py
2022-05-18T04:13:08.9770785Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmparpt9n6v
2022-05-18T04:13:08.9772442Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmparpt9n6v/_remote_module_non_scriptable.py
2022-05-18T04:13:08.9865752Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeixo29ao
2022-05-18T04:13:08.9866695Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeixo29ao/_remote_module_non_scriptable.py
2022-05-18T04:13:09.1994226Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:13:09.2222542Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:13:09.2324690Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:13:09.2418275Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:13:09.4535572Z ok (1.436s)
2022-05-18T04:13:09.4535970Z 
2022-05-18T04:13:09.4536565Z ----------------------------------------------------------------------
2022-05-18T04:13:09.4536847Z Ran 1 test in 1.437s
2022-05-18T04:13:09.4536972Z 
2022-05-18T04:13:09.4537079Z OK
2022-05-18T04:13:09.4537159Z 
2022-05-18T04:13:09.4537254Z Generating XML reports...
2022-05-18T04:13:09.4571523Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220518041308.xml
2022-05-18T04:13:10.2301719Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp35ysm8_b
2022-05-18T04:13:10.2302451Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp35ysm8_b/_remote_module_non_scriptable.py
2022-05-18T04:13:10.4860585Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:13:10.4870980Z 
2022-05-18T04:13:10.4871260Z Running tests...
2022-05-18T04:13:10.4871927Z ----------------------------------------------------------------------
2022-05-18T04:13:10.8105861Z   test_create_remote_module_from_module_rref (__main__.TensorPipeThreeWorkersRemoteModuleTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2966
2022-05-18T04:13:10.8128218Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2967
2022-05-18T04:13:10.8151312Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2968
2022-05-18T04:13:11.3983874Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0u3e6vbw
2022-05-18T04:13:11.3984658Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0u3e6vbw/_remote_module_non_scriptable.py
2022-05-18T04:13:11.4057304Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7xy2t4e0
2022-05-18T04:13:11.4058609Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7xy2t4e0/_remote_module_non_scriptable.py
2022-05-18T04:13:11.4202211Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0qmfz3qi
2022-05-18T04:13:11.4203283Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0qmfz3qi/_remote_module_non_scriptable.py
2022-05-18T04:13:11.6471535Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:13:11.6523411Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:13:11.6695522Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:13:12.1187313Z ok (1.631s)
2022-05-18T04:13:12.1187526Z 
2022-05-18T04:13:12.1188025Z ----------------------------------------------------------------------
2022-05-18T04:13:12.1188450Z Ran 1 test in 1.632s
2022-05-18T04:13:12.1188600Z 
2022-05-18T04:13:12.1188682Z OK
2022-05-18T04:13:12.1188776Z 
2022-05-18T04:13:12.1188875Z Generating XML reports...
2022-05-18T04:13:12.1223307Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeThreeWorkersRemoteModuleTest-20220518041310.xml
2022-05-18T04:13:12.9345667Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdacuhawe
2022-05-18T04:13:12.9346431Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdacuhawe/_remote_module_non_scriptable.py
2022-05-18T04:13:13.1914716Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:13:13.1924591Z 
2022-05-18T04:13:13.1924885Z Running tests...
2022-05-18T04:13:13.1925595Z ----------------------------------------------------------------------
2022-05-18T04:13:13.5162692Z   test_send_remote_module_over_the_wire (__main__.TensorPipeThreeWorkersRemoteModuleTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3133
2022-05-18T04:13:13.5184774Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3134
2022-05-18T04:13:13.5208788Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3135
2022-05-18T04:13:14.1063806Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_bqschu1
2022-05-18T04:13:14.1064637Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_bqschu1/_remote_module_non_scriptable.py
2022-05-18T04:13:14.1127866Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpigzpajqu
2022-05-18T04:13:14.1128764Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpigzpajqu/_remote_module_non_scriptable.py
2022-05-18T04:13:14.1215171Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2ud4g2ai
2022-05-18T04:13:14.1216497Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2ud4g2ai/_remote_module_non_scriptable.py
2022-05-18T04:13:14.3560023Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:13:14.3610649Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:13:14.3718118Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:13:14.7242073Z ok (1.531s)
2022-05-18T04:13:14.7242274Z 
2022-05-18T04:13:14.7242678Z ----------------------------------------------------------------------
2022-05-18T04:13:14.7242957Z Ran 1 test in 1.532s
2022-05-18T04:13:14.7243059Z 
2022-05-18T04:13:14.7243123Z OK
2022-05-18T04:13:14.7243217Z 
2022-05-18T04:13:14.7243311Z Generating XML reports...
2022-05-18T04:13:14.7278003Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeThreeWorkersRemoteModuleTest-20220518041313.xml
2022-05-18T04:13:15.5178302Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphknna89x
2022-05-18T04:13:15.5179469Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphknna89x/_remote_module_non_scriptable.py
2022-05-18T04:13:15.7754478Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent
2022-05-18T04:13:15.7764344Z 
2022-05-18T04:13:15.7764778Z Running tests...
2022-05-18T04:13:15.7765265Z ----------------------------------------------------------------------
2022-05-18T04:13:16.1059832Z   test_send_remote_module_over_the_wire_script_not_supported (__main__.TensorPipeThreeWorkersRemoteModuleTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3300
2022-05-18T04:13:16.1082555Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3301
2022-05-18T04:13:16.1106468Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3302
2022-05-18T04:13:16.7120868Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm9kbpd21
2022-05-18T04:13:16.7121658Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm9kbpd21/_remote_module_non_scriptable.py
2022-05-18T04:13:16.7180289Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpezlk6teu
2022-05-18T04:13:16.7181661Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpezlk6teu/_remote_module_non_scriptable.py
2022-05-18T04:13:16.7238042Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmf_n75u5
2022-05-18T04:13:16.7239297Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmf_n75u5/_remote_module_non_scriptable.py
2022-05-18T04:13:16.9609984Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:13:16.9676425Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:13:16.9718108Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:13:17.1333528Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmf_n75u5/_remote_module___torch___torch_testing__internal_distributed_nn_api_remote_module_test_MyModuleInterface.py
2022-05-18T04:13:17.1334854Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm9kbpd21/_remote_module___torch___torch_testing__internal_distributed_nn_api_remote_module_test_MyModuleInterface.py
2022-05-18T04:13:17.1382539Z INFO:torch.distributed.nn.jit.instantiator:Skipped writing /tmp/tmpmf_n75u5/_remote_module___torch___torch_testing__internal_distributed_nn_api_remote_module_test_MyModuleInterface.py
2022-05-18T04:13:17.4141262Z ok (1.637s)
2022-05-18T04:13:17.4141490Z 
2022-05-18T04:13:17.4141958Z ----------------------------------------------------------------------
2022-05-18T04:13:17.4142354Z Ran 1 test in 1.638s
2022-05-18T04:13:17.4142539Z 
2022-05-18T04:13:17.4142638Z OK
2022-05-18T04:13:17.4142786Z 
2022-05-18T04:13:17.4143109Z Generating XML reports...
2022-05-18T04:13:17.4177817Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeThreeWorkersRemoteModuleTest-20220518041315.xml
2022-05-18T04:13:17.7535448Z Running distributed/test_c10d_common ... [2022-05-18 04:13:17.753126]
2022-05-18T04:13:17.7536433Z Executing ['/opt/conda/bin/python', 'distributed/test_c10d_common.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-05-18 04:13:17.753208]
2022-05-18T04:13:18.3355238Z test_debug_level (__main__.CommTest)
2022-05-18T04:13:18.3355808Z test_multi_limit_multi_dtype (__main__.ComputeBucketAssignmentTest)
2022-05-18T04:13:18.3356201Z test_multi_limit_single_dtype (__main__.ComputeBucketAssignmentTest)
2022-05-18T04:13:18.3356533Z test_single_limit_multi_dtype (__main__.ComputeBucketAssignmentTest)
2022-05-18T04:13:18.3356851Z test_single_limit_single_dtype (__main__.ComputeBucketAssignmentTest)
2022-05-18T04:13:18.3357161Z test_backend_class_attr (__main__.PythonProcessGroupExtensionTest)
2022-05-18T04:13:18.3357482Z test_collectives (__main__.PythonProcessGroupExtensionTest)
2022-05-18T04:13:18.3358021Z test_get_backend_name (__main__.PythonProcessGroupExtensionTest)
2022-05-18T04:13:18.3358315Z test_send_recv (__main__.PythonProcessGroupExtensionTest)
2022-05-18T04:13:18.9072728Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common
2022-05-18T04:13:18.9081964Z 
2022-05-18T04:13:18.9082097Z Running tests...
2022-05-18T04:13:18.9082861Z ----------------------------------------------------------------------
2022-05-18T04:13:19.1938420Z   test_debug_level (__main__.CommTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3477
2022-05-18T04:13:19.1960666Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3478
2022-05-18T04:13:19.7663738Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:13:19.7907309Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:13:19.9985341Z ok (1.090s)
2022-05-18T04:13:19.9985584Z 
2022-05-18T04:13:19.9986114Z ----------------------------------------------------------------------
2022-05-18T04:13:19.9986467Z Ran 1 test in 1.090s
2022-05-18T04:13:19.9986585Z 
2022-05-18T04:13:19.9986645Z OK
2022-05-18T04:13:19.9986744Z 
2022-05-18T04:13:19.9986838Z Generating XML reports...
2022-05-18T04:13:20.0023117Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-CommTest-20220518041318.xml
2022-05-18T04:13:20.7706097Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common
2022-05-18T04:13:20.7715281Z 
2022-05-18T04:13:20.7715416Z Running tests...
2022-05-18T04:13:20.7716338Z ----------------------------------------------------------------------
2022-05-18T04:13:21.0516750Z   test_multi_limit_multi_dtype (__main__.ComputeBucketAssignmentTest) ... ok (0.280s)
2022-05-18T04:13:21.0517171Z 
2022-05-18T04:13:21.0517523Z ----------------------------------------------------------------------
2022-05-18T04:13:21.0517758Z Ran 1 test in 0.280s
2022-05-18T04:13:21.0517890Z 
2022-05-18T04:13:21.0517952Z OK
2022-05-18T04:13:21.0518049Z 
2022-05-18T04:13:21.0518138Z Generating XML reports...
2022-05-18T04:13:21.0541302Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-ComputeBucketAssignmentTest-20220518041320.xml
2022-05-18T04:13:21.7936142Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common
2022-05-18T04:13:21.7946261Z 
2022-05-18T04:13:21.7946634Z Running tests...
2022-05-18T04:13:21.7947056Z ----------------------------------------------------------------------
2022-05-18T04:13:22.0762582Z   test_multi_limit_single_dtype (__main__.ComputeBucketAssignmentTest) ... ok (0.281s)
2022-05-18T04:13:22.0762977Z 
2022-05-18T04:13:22.0763444Z ----------------------------------------------------------------------
2022-05-18T04:13:22.0763870Z Ran 1 test in 0.281s
2022-05-18T04:13:22.0764082Z 
2022-05-18T04:13:22.0764190Z OK
2022-05-18T04:13:22.0764357Z 
2022-05-18T04:13:22.0764526Z Generating XML reports...
2022-05-18T04:13:22.0788131Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-ComputeBucketAssignmentTest-20220518041321.xml
2022-05-18T04:13:22.8237803Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common
2022-05-18T04:13:22.8247137Z 
2022-05-18T04:13:22.8247275Z Running tests...
2022-05-18T04:13:22.8247690Z ----------------------------------------------------------------------
2022-05-18T04:13:23.1061172Z   test_single_limit_multi_dtype (__main__.ComputeBucketAssignmentTest) ... ok (0.281s)
2022-05-18T04:13:23.1061551Z 
2022-05-18T04:13:23.1061964Z ----------------------------------------------------------------------
2022-05-18T04:13:23.1062203Z Ran 1 test in 0.281s
2022-05-18T04:13:23.1062320Z 
2022-05-18T04:13:23.1062382Z OK
2022-05-18T04:13:23.1062475Z 
2022-05-18T04:13:23.1062565Z Generating XML reports...
2022-05-18T04:13:23.1086209Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-ComputeBucketAssignmentTest-20220518041322.xml
2022-05-18T04:13:23.8472221Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common
2022-05-18T04:13:23.8481712Z 
2022-05-18T04:13:23.8482059Z Running tests...
2022-05-18T04:13:23.8482673Z ----------------------------------------------------------------------
2022-05-18T04:13:24.1270716Z   test_single_limit_single_dtype (__main__.ComputeBucketAssignmentTest) ... ok (0.279s)
2022-05-18T04:13:24.1271096Z 
2022-05-18T04:13:24.1271593Z ----------------------------------------------------------------------
2022-05-18T04:13:24.1272072Z Ran 1 test in 0.279s
2022-05-18T04:13:24.1272207Z 
2022-05-18T04:13:24.1272270Z OK
2022-05-18T04:13:24.1272362Z 
2022-05-18T04:13:24.1272434Z Generating XML reports...
2022-05-18T04:13:24.1294902Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-ComputeBucketAssignmentTest-20220518041323.xml
2022-05-18T04:13:24.8668867Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common
2022-05-18T04:13:24.8678117Z 
2022-05-18T04:13:24.8678503Z Running tests...
2022-05-18T04:13:24.8678891Z ----------------------------------------------------------------------
2022-05-18T04:13:25.1521179Z   test_backend_class_attr (__main__.PythonProcessGroupExtensionTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3550
2022-05-18T04:13:25.1542359Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3551
2022-05-18T04:13:25.1565057Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3552
2022-05-18T04:13:25.1588873Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3553
2022-05-18T04:13:25.7369804Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:13:25.7493232Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:13:25.8000814Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:13:25.8014757Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:13:25.9618717Z ok (1.094s)
2022-05-18T04:13:25.9618983Z 
2022-05-18T04:13:25.9619508Z ----------------------------------------------------------------------
2022-05-18T04:13:25.9619939Z Ran 1 test in 1.094s
2022-05-18T04:13:25.9620056Z 
2022-05-18T04:13:25.9620124Z OK
2022-05-18T04:13:25.9620216Z 
2022-05-18T04:13:25.9620311Z Generating XML reports...
2022-05-18T04:13:25.9654152Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-PythonProcessGroupExtensionTest-20220518041324.xml
2022-05-18T04:13:26.7300156Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common
2022-05-18T04:13:26.7309329Z 
2022-05-18T04:13:26.7309422Z Running tests...
2022-05-18T04:13:26.7310364Z ----------------------------------------------------------------------
2022-05-18T04:13:27.0157308Z   test_collectives (__main__.PythonProcessGroupExtensionTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3605
2022-05-18T04:13:27.0178563Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3606
2022-05-18T04:13:27.0200783Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3607
2022-05-18T04:13:27.0225085Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3608
2022-05-18T04:13:27.6857945Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:13:27.6967369Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:13:27.7491012Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:13:27.7498211Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:13:27.7518866Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:13:27.7526692Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:13:28.6870737Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T04:13:28.6907091Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:13:28.6908014Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:13:28.6930065Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:13:28.6972364Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:13:28.7004330Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:13:28.9271329Z ok (2.196s)
2022-05-18T04:13:28.9271516Z 
2022-05-18T04:13:28.9271874Z ----------------------------------------------------------------------
2022-05-18T04:13:28.9272191Z Ran 1 test in 2.196s
2022-05-18T04:13:28.9272307Z 
2022-05-18T04:13:28.9272357Z OK
2022-05-18T04:13:28.9272450Z 
2022-05-18T04:13:28.9272545Z Generating XML reports...
2022-05-18T04:13:28.9309084Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-PythonProcessGroupExtensionTest-20220518041326.xml
2022-05-18T04:13:29.6982330Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common
2022-05-18T04:13:29.6992010Z 
2022-05-18T04:13:29.6992143Z Running tests...
2022-05-18T04:13:29.6992733Z ----------------------------------------------------------------------
2022-05-18T04:13:29.9832946Z   test_get_backend_name (__main__.PythonProcessGroupExtensionTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3665
2022-05-18T04:13:29.9854533Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3666
2022-05-18T04:13:29.9877363Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3667
2022-05-18T04:13:29.9900054Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3668
2022-05-18T04:13:30.5655840Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:13:30.6227713Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:13:30.6410587Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:13:30.6446022Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:13:30.7929392Z ok (1.093s)
2022-05-18T04:13:30.7929658Z 
2022-05-18T04:13:30.7929979Z ----------------------------------------------------------------------
2022-05-18T04:13:30.7930247Z Ran 1 test in 1.094s
2022-05-18T04:13:30.7930363Z 
2022-05-18T04:13:30.7930427Z OK
2022-05-18T04:13:30.7930505Z 
2022-05-18T04:13:30.7930599Z Generating XML reports...
2022-05-18T04:13:30.7965979Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-PythonProcessGroupExtensionTest-20220518041329.xml
2022-05-18T04:13:31.5525309Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common
2022-05-18T04:13:31.5534929Z 
2022-05-18T04:13:31.5535056Z Running tests...
2022-05-18T04:13:31.5535494Z ----------------------------------------------------------------------
2022-05-18T04:13:31.8395885Z   test_send_recv (__main__.PythonProcessGroupExtensionTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3720
2022-05-18T04:13:31.8417575Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3721
2022-05-18T04:13:31.8439551Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3722
2022-05-18T04:13:31.8463615Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3723
2022-05-18T04:13:32.5201446Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:13:32.5685740Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:13:32.5774887Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:13:32.5782168Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:13:32.5819213Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:13:32.5826857Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:13:33.5212901Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T04:13:33.5311111Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:13:33.5312086Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:13:33.5314682Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:13:33.5331469Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:13:33.5394623Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:13:33.7509576Z ok (2.197s)
2022-05-18T04:13:33.7509826Z 
2022-05-18T04:13:33.7510265Z ----------------------------------------------------------------------
2022-05-18T04:13:33.7510523Z Ran 1 test in 2.197s
2022-05-18T04:13:33.7510645Z 
2022-05-18T04:13:33.7510724Z OK
2022-05-18T04:13:33.7510841Z 
2022-05-18T04:13:33.7510952Z Generating XML reports...
2022-05-18T04:13:33.7543662Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-PythonProcessGroupExtensionTest-20220518041331.xml
2022-05-18T04:13:34.0755271Z Running distributed/test_c10d_gloo ... [2022-05-18 04:13:34.075121]
2022-05-18T04:13:34.0755843Z Executing ['/opt/conda/bin/python', 'distributed/test_c10d_gloo.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-05-18 04:13:34.075202]
2022-05-18T04:13:34.6576495Z <unittest.suite.TestSuite tests=[<__main__.CommTest testMethod=test_broadcast_coalesced_gloo_cpu>, <__main__.CommTest testMethod=test_broadcast_coalesced_gloo_cuda>, <__main__.CommTest testMethod=test_gloo_barrier_device_ids>, <__main__.CommTest testMethod=test_gloo_warn_not_in_group>, <__main__.CommTest testMethod=test_sequence_num_incremented_gloo_default>, <__main__.CommTest testMethod=test_sequence_num_incremented_gloo_subgroup>, <__main__.CommTest testMethod=test_sequence_num_set_default_pg_gloo>, <__main__.CommTest testMethod=test_sequence_num_set_gloo_new_group>]>
2022-05-18T04:13:34.6577773Z test_broadcast_coalesced_gloo_cpu (__main__.CommTest)
2022-05-18T04:13:34.6578216Z test_broadcast_coalesced_gloo_cuda (__main__.CommTest)
2022-05-18T04:13:34.6578675Z test_gloo_barrier_device_ids (__main__.CommTest)
2022-05-18T04:13:34.6579055Z test_gloo_warn_not_in_group (__main__.CommTest)
2022-05-18T04:13:34.6579454Z test_sequence_num_incremented_gloo_default (__main__.CommTest)
2022-05-18T04:13:34.6579892Z test_sequence_num_incremented_gloo_subgroup (__main__.CommTest)
2022-05-18T04:13:34.6580281Z test_sequence_num_set_default_pg_gloo (__main__.CommTest)
2022-05-18T04:13:34.6580531Z test_sequence_num_set_gloo_new_group (__main__.CommTest)
2022-05-18T04:13:34.6585850Z <unittest.suite.TestSuite tests=[<__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_dynamic_module>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_dynamic_weight_sharing>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_once_use_reentrant_False>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_once_use_reentrant_True>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_twice_static_graph_use_reentrant_False>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_twice_static_graph_use_reentrant_True>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_twice_use_reentrant_False>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_twice_use_reentrant_True>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_twice_weight_sharing>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_unused_params_use_reentrant_False>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_unused_params_use_reentrant_True>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_weight_sharing_use_reentrant_False>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_weight_sharing_use_reentrant_True>, <__main__.DistributedDataParallelTest testMethod=test_ddp_comm_hook_future_passing_cpu>, <__main__.DistributedDataParallelTest testMethod=test_ddp_comm_hook_future_passing_gpu_gloo>, <__main__.DistributedDataParallelTest testMethod=test_ddp_comm_hook_register_just_once>, <__main__.DistributedDataParallelTest testMethod=test_ddp_comm_hook_sparse_gradients>, <__main__.DistributedDataParallelTest testMethod=test_ddp_invalid_comm_hook_init>, <__main__.DistributedDataParallelTest testMethod=test_ddp_invalid_comm_hook_return_type>, <__main__.DistributedDataParallelTest testMethod=test_find_unused_parameters_when_unused_parameters_empty>, <__main__.DistributedDataParallelTest testMethod=test_global_local_unused_params_grad>, <__main__.DistributedDataParallelTest testMethod=test_global_local_unused_params_grad_with_grad_is_view>, <__main__.DistributedDataParallelTest testMethod=test_global_local_unused_params_grad_with_static_graph>, <__main__.DistributedDataParallelTest testMethod=test_gloo_backend_1gpu_module_device_ids_integer_list>, <__main__.DistributedDataParallelTest testMethod=test_gloo_backend_1gpu_module_device_ids_torch_device_list>, <__main__.DistributedDataParallelTest testMethod=test_gloo_backend_2gpu_module>, <__main__.DistributedDataParallelTest testMethod=test_gloo_backend_4gpu_module>, <__main__.DistributedDataParallelTest testMethod=test_gloo_backend_cpu_module>, <__main__.DistributedDataParallelTest testMethod=test_gloo_backend_cpu_module_grad_is_view>, <__main__.DistributedDataParallelTest testMethod=test_ignored_output>, <__main__.DistributedDataParallelTest testMethod=test_ignored_output_with_unused_parameters>, <__main__.DistributedDataParallelTest testMethod=test_invalid_powerSGD_state>, <__main__.DistributedDataParallelTest testMethod=test_save_load_checkpoint>, <__main__.DistributedDataParallelTest testMethod=test_sparse_gradients>, <__main__.DistributedDataParallelTest testMethod=test_sparse_gradients_grad_is_view>, <__main__.DistributedDataParallelTest testMethod=test_sync_batch_norm_empty_input>, <__main__.DistributedDataParallelTest testMethod=test_sync_batch_norm_only_empty_input>]>
2022-05-18T04:13:34.6590451Z test_ddp_checkpointing_dynamic_module (__main__.DistributedDataParallelTest)
2022-05-18T04:13:34.6590849Z test_ddp_checkpointing_dynamic_weight_sharing (__main__.DistributedDataParallelTest)
2022-05-18T04:13:34.6591214Z test_ddp_checkpointing_once_use_reentrant_False (__main__.DistributedDataParallelTest)
2022-05-18T04:13:34.6591620Z test_ddp_checkpointing_once_use_reentrant_True (__main__.DistributedDataParallelTest)
2022-05-18T04:13:34.6591981Z test_ddp_checkpointing_twice_static_graph_use_reentrant_False (__main__.DistributedDataParallelTest)
2022-05-18T04:13:34.6592420Z test_ddp_checkpointing_twice_static_graph_use_reentrant_True (__main__.DistributedDataParallelTest)
2022-05-18T04:13:34.6592790Z test_ddp_checkpointing_twice_use_reentrant_False (__main__.DistributedDataParallelTest)
2022-05-18T04:13:34.6593233Z test_ddp_checkpointing_twice_use_reentrant_True (__main__.DistributedDataParallelTest)
2022-05-18T04:13:34.6593628Z test_ddp_checkpointing_twice_weight_sharing (__main__.DistributedDataParallelTest)
2022-05-18T04:13:34.6594017Z test_ddp_checkpointing_unused_params_use_reentrant_False (__main__.DistributedDataParallelTest)
2022-05-18T04:13:34.6594425Z test_ddp_checkpointing_unused_params_use_reentrant_True (__main__.DistributedDataParallelTest)
2022-05-18T04:13:34.6594796Z test_ddp_checkpointing_weight_sharing_use_reentrant_False (__main__.DistributedDataParallelTest)
2022-05-18T04:13:34.6595219Z test_ddp_checkpointing_weight_sharing_use_reentrant_True (__main__.DistributedDataParallelTest)
2022-05-18T04:13:34.6595571Z test_ddp_comm_hook_future_passing_cpu (__main__.DistributedDataParallelTest)
2022-05-18T04:13:34.6595939Z test_ddp_comm_hook_future_passing_gpu_gloo (__main__.DistributedDataParallelTest)
2022-05-18T04:13:34.6596267Z test_ddp_comm_hook_register_just_once (__main__.DistributedDataParallelTest)
2022-05-18T04:13:34.6596599Z test_ddp_comm_hook_sparse_gradients (__main__.DistributedDataParallelTest)
2022-05-18T04:13:34.6596973Z test_ddp_invalid_comm_hook_init (__main__.DistributedDataParallelTest)
2022-05-18T04:13:34.6597282Z test_ddp_invalid_comm_hook_return_type (__main__.DistributedDataParallelTest)
2022-05-18T04:13:34.6597689Z test_find_unused_parameters_when_unused_parameters_empty (__main__.DistributedDataParallelTest)
2022-05-18T04:13:34.6598044Z test_global_local_unused_params_grad (__main__.DistributedDataParallelTest)
2022-05-18T04:13:34.6598440Z test_global_local_unused_params_grad_with_grad_is_view (__main__.DistributedDataParallelTest)
2022-05-18T04:13:34.6598793Z test_global_local_unused_params_grad_with_static_graph (__main__.DistributedDataParallelTest)
2022-05-18T04:13:34.6599205Z test_gloo_backend_1gpu_module_device_ids_integer_list (__main__.DistributedDataParallelTest)
2022-05-18T04:13:34.6599568Z test_gloo_backend_1gpu_module_device_ids_torch_device_list (__main__.DistributedDataParallelTest)
2022-05-18T04:13:34.6599960Z test_gloo_backend_2gpu_module (__main__.DistributedDataParallelTest)
2022-05-18T04:13:34.6600262Z test_gloo_backend_4gpu_module (__main__.DistributedDataParallelTest)
2022-05-18T04:13:34.6600620Z test_gloo_backend_cpu_module (__main__.DistributedDataParallelTest)
2022-05-18T04:13:34.6600947Z test_gloo_backend_cpu_module_grad_is_view (__main__.DistributedDataParallelTest)
2022-05-18T04:13:34.6601294Z test_ignored_output (__main__.DistributedDataParallelTest)
2022-05-18T04:13:34.6601617Z test_ignored_output_with_unused_parameters (__main__.DistributedDataParallelTest)
2022-05-18T04:13:34.6601951Z test_invalid_powerSGD_state (__main__.DistributedDataParallelTest)
2022-05-18T04:13:34.6602281Z test_save_load_checkpoint (__main__.DistributedDataParallelTest)
2022-05-18T04:13:34.6602581Z test_sparse_gradients (__main__.DistributedDataParallelTest)
2022-05-18T04:13:34.6602937Z test_sparse_gradients_grad_is_view (__main__.DistributedDataParallelTest)
2022-05-18T04:13:34.6603261Z test_sync_batch_norm_empty_input (__main__.DistributedDataParallelTest)
2022-05-18T04:13:34.6603616Z test_sync_batch_norm_only_empty_input (__main__.DistributedDataParallelTest)
2022-05-18T04:13:34.6603893Z <unittest.suite.TestSuite tests=[]>
2022-05-18T04:13:34.6608373Z <unittest.suite.TestSuite tests=[<__main__.ProcessGroupGlooTest testMethod=test_allgather_basics>, <__main__.ProcessGroupGlooTest testMethod=test_allgather_basics_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_allgather_checks>, <__main__.ProcessGroupGlooTest testMethod=test_allgather_coalesced_async>, <__main__.ProcessGroupGlooTest testMethod=test_allgather_coalesced_checks>, <__main__.ProcessGroupGlooTest testMethod=test_allgather_noncontiguous_input>, <__main__.ProcessGroupGlooTest testMethod=test_allgather_stress>, <__main__.ProcessGroupGlooTest testMethod=test_allgather_stress_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_allreduce_basics>, <__main__.ProcessGroupGlooTest testMethod=test_allreduce_basics_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_allreduce_basics_cuda_using_work_api>, <__main__.ProcessGroupGlooTest testMethod=test_allreduce_basics_using_work_api>, <__main__.ProcessGroupGlooTest testMethod=test_allreduce_checks>, <__main__.ProcessGroupGlooTest testMethod=test_allreduce_coalesced_async>, <__main__.ProcessGroupGlooTest testMethod=test_allreduce_coalesced_basics>, <__main__.ProcessGroupGlooTest testMethod=test_allreduce_coalesced_checks>, <__main__.ProcessGroupGlooTest testMethod=test_allreduce_coalesced_checks_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_allreduce_coalesced_stress>, <__main__.ProcessGroupGlooTest testMethod=test_allreduce_stress>, <__main__.ProcessGroupGlooTest testMethod=test_allreduce_stress_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_barrier_implies_wait>, <__main__.ProcessGroupGlooTest testMethod=test_broadcast_basics>, <__main__.ProcessGroupGlooTest testMethod=test_broadcast_basics_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_broadcast_checks>, <__main__.ProcessGroupGlooTest testMethod=test_broadcast_stress>, <__main__.ProcessGroupGlooTest testMethod=test_broadcast_stress_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_empty_tensors>, <__main__.ProcessGroupGlooTest testMethod=test_gather_basics>, <__main__.ProcessGroupGlooTest testMethod=test_gather_basics_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_gather_checks>, <__main__.ProcessGroupGlooTest testMethod=test_gather_noncontiguous_input>, <__main__.ProcessGroupGlooTest testMethod=test_gather_stress>, <__main__.ProcessGroupGlooTest testMethod=test_gather_stress_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_multi_device_constructor>, <__main__.ProcessGroupGlooTest testMethod=test_reduce_basics>, <__main__.ProcessGroupGlooTest testMethod=test_reduce_basics_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_reduce_checks>, <__main__.ProcessGroupGlooTest testMethod=test_reduce_stress>, <__main__.ProcessGroupGlooTest testMethod=test_reduce_stress_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_round_robin>, <__main__.ProcessGroupGlooTest testMethod=test_round_robin_create_destroy>, <__main__.ProcessGroupGlooTest testMethod=test_scatter_basics>, <__main__.ProcessGroupGlooTest testMethod=test_scatter_basics_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_scatter_checks>, <__main__.ProcessGroupGlooTest testMethod=test_scatter_stress>, <__main__.ProcessGroupGlooTest testMethod=test_scatter_stress_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_send_recv_all_to_all>, <__main__.ProcessGroupGlooTest testMethod=test_sparse_allreduce_basics>, <__main__.ProcessGroupGlooTest testMethod=test_sparse_allreduce_basics_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_sparse_allreduce_checks>]>
2022-05-18T04:13:34.6612589Z test_allgather_basics (__main__.ProcessGroupGlooTest)
2022-05-18T04:13:34.6612924Z test_allgather_basics_cuda (__main__.ProcessGroupGlooTest)
2022-05-18T04:13:34.6613199Z test_allgather_checks (__main__.ProcessGroupGlooTest)
2022-05-18T04:13:34.6613469Z test_allgather_coalesced_async (__main__.ProcessGroupGlooTest)
2022-05-18T04:13:34.6613756Z test_allgather_coalesced_checks (__main__.ProcessGroupGlooTest)
2022-05-18T04:13:34.6614057Z test_allgather_noncontiguous_input (__main__.ProcessGroupGlooTest)
2022-05-18T04:13:34.6614329Z test_allgather_stress (__main__.ProcessGroupGlooTest)
2022-05-18T04:13:34.6614604Z test_allgather_stress_cuda (__main__.ProcessGroupGlooTest)
2022-05-18T04:13:34.6614924Z test_allreduce_basics (__main__.ProcessGroupGlooTest)
2022-05-18T04:13:34.6615194Z test_allreduce_basics_cuda (__main__.ProcessGroupGlooTest)
2022-05-18T04:13:34.6615477Z test_allreduce_basics_cuda_using_work_api (__main__.ProcessGroupGlooTest)
2022-05-18T04:13:34.6615783Z test_allreduce_basics_using_work_api (__main__.ProcessGroupGlooTest)
2022-05-18T04:13:34.6616067Z test_allreduce_checks (__main__.ProcessGroupGlooTest)
2022-05-18T04:13:34.6616332Z test_allreduce_coalesced_async (__main__.ProcessGroupGlooTest)
2022-05-18T04:13:34.6616659Z test_allreduce_coalesced_basics (__main__.ProcessGroupGlooTest)
2022-05-18T04:13:34.6616975Z test_allreduce_coalesced_checks (__main__.ProcessGroupGlooTest)
2022-05-18T04:13:34.6617304Z test_allreduce_coalesced_checks_cuda (__main__.ProcessGroupGlooTest)
2022-05-18T04:13:34.6617590Z test_allreduce_coalesced_stress (__main__.ProcessGroupGlooTest)
2022-05-18T04:13:34.6617866Z test_allreduce_stress (__main__.ProcessGroupGlooTest)
2022-05-18T04:13:34.6618137Z test_allreduce_stress_cuda (__main__.ProcessGroupGlooTest)
2022-05-18T04:13:34.6618399Z test_barrier_implies_wait (__main__.ProcessGroupGlooTest)
2022-05-18T04:13:34.6618713Z test_broadcast_basics (__main__.ProcessGroupGlooTest)
2022-05-18T04:13:34.6618989Z test_broadcast_basics_cuda (__main__.ProcessGroupGlooTest)
2022-05-18T04:13:34.6619245Z test_broadcast_checks (__main__.ProcessGroupGlooTest)
2022-05-18T04:13:34.6619517Z test_broadcast_stress (__main__.ProcessGroupGlooTest)
2022-05-18T04:13:34.6619787Z test_broadcast_stress_cuda (__main__.ProcessGroupGlooTest)
2022-05-18T04:13:34.6620056Z test_empty_tensors (__main__.ProcessGroupGlooTest)
2022-05-18T04:13:34.6620302Z test_gather_basics (__main__.ProcessGroupGlooTest)
2022-05-18T04:13:34.6620570Z test_gather_basics_cuda (__main__.ProcessGroupGlooTest)
2022-05-18T04:13:34.6620832Z test_gather_checks (__main__.ProcessGroupGlooTest)
2022-05-18T04:13:34.6621096Z test_gather_noncontiguous_input (__main__.ProcessGroupGlooTest)
2022-05-18T04:13:34.6621369Z test_gather_stress (__main__.ProcessGroupGlooTest)
2022-05-18T04:13:34.6621632Z test_gather_stress_cuda (__main__.ProcessGroupGlooTest)
2022-05-18T04:13:34.6621898Z test_multi_device_constructor (__main__.ProcessGroupGlooTest)
2022-05-18T04:13:34.6622168Z test_reduce_basics (__main__.ProcessGroupGlooTest)
2022-05-18T04:13:34.6622433Z test_reduce_basics_cuda (__main__.ProcessGroupGlooTest)
2022-05-18T04:13:34.6622694Z test_reduce_checks (__main__.ProcessGroupGlooTest)
2022-05-18T04:13:34.6623067Z test_reduce_stress (__main__.ProcessGroupGlooTest)
2022-05-18T04:13:34.6623417Z test_reduce_stress_cuda (__main__.ProcessGroupGlooTest)
2022-05-18T04:13:34.6623679Z test_round_robin (__main__.ProcessGroupGlooTest)
2022-05-18T04:13:34.6623941Z test_round_robin_create_destroy (__main__.ProcessGroupGlooTest)
2022-05-18T04:13:34.6624214Z test_scatter_basics (__main__.ProcessGroupGlooTest)
2022-05-18T04:13:34.6624480Z test_scatter_basics_cuda (__main__.ProcessGroupGlooTest)
2022-05-18T04:13:34.6624734Z test_scatter_checks (__main__.ProcessGroupGlooTest)
2022-05-18T04:13:34.6624988Z test_scatter_stress (__main__.ProcessGroupGlooTest)
2022-05-18T04:13:34.6625253Z test_scatter_stress_cuda (__main__.ProcessGroupGlooTest)
2022-05-18T04:13:34.6625514Z test_send_recv_all_to_all (__main__.ProcessGroupGlooTest)
2022-05-18T04:13:34.6625845Z test_sparse_allreduce_basics (__main__.ProcessGroupGlooTest)
2022-05-18T04:13:34.6626133Z test_sparse_allreduce_basics_cuda (__main__.ProcessGroupGlooTest)
2022-05-18T04:13:34.6626420Z test_sparse_allreduce_checks (__main__.ProcessGroupGlooTest)
2022-05-18T04:13:34.6627067Z <unittest.suite.TestSuite tests=[<__main__.ReducerTest testMethod=test_forward_backward>, <__main__.ReducerTest testMethod=test_forward_backward_optimizer>, <__main__.ReducerTest testMethod=test_forward_backward_unused_parameters>, <__main__.ReducerTest testMethod=test_multi_dtype_multi_bucket>, <__main__.ReducerTest testMethod=test_multi_dtype_single_bucket>, <__main__.ReducerTest testMethod=test_single_dtype_single_bucket>]>
2022-05-18T04:13:34.6627668Z test_forward_backward (__main__.ReducerTest)
2022-05-18T04:13:34.6627916Z test_forward_backward_optimizer (__main__.ReducerTest)
2022-05-18T04:13:34.6628183Z test_forward_backward_unused_parameters (__main__.ReducerTest)
2022-05-18T04:13:34.6628433Z test_multi_dtype_multi_bucket (__main__.ReducerTest)
2022-05-18T04:13:34.6628682Z test_multi_dtype_single_bucket (__main__.ReducerTest)
2022-05-18T04:13:34.6628932Z test_single_dtype_single_bucket (__main__.ReducerTest)
2022-05-18T04:13:34.6629238Z <unittest.suite.TestSuite tests=[<__main__.RendezvousEnvTest testMethod=test_logging_init>]>
2022-05-18T04:13:34.6629624Z test_logging_init (__main__.RendezvousEnvTest)
2022-05-18T04:13:34.6629860Z <unittest.suite.TestSuite tests=[]>
2022-05-18T04:13:34.6630216Z <unittest.suite.TestSuite tests=[<__main__.TimeoutTest testMethod=test_default_store_timeout_gloo>]>
2022-05-18T04:13:34.6630516Z test_default_store_timeout_gloo (__main__.TimeoutTest)
2022-05-18T04:13:35.2321024Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:13:35.2330026Z 
2022-05-18T04:13:35.2330143Z Running tests...
2022-05-18T04:13:35.2330605Z ----------------------------------------------------------------------
2022-05-18T04:13:35.5179588Z   test_broadcast_coalesced_gloo_cpu (__main__.CommTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3790
2022-05-18T04:13:35.5200720Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3791
2022-05-18T04:13:36.1107499Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:13:36.1190992Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:13:36.3225505Z ok (1.089s)
2022-05-18T04:13:36.3225781Z 
2022-05-18T04:13:36.3226228Z ----------------------------------------------------------------------
2022-05-18T04:13:36.3226499Z Ran 1 test in 1.089s
2022-05-18T04:13:36.3226619Z 
2022-05-18T04:13:36.3226682Z OK
2022-05-18T04:13:36.3226777Z 
2022-05-18T04:13:36.3226858Z Generating XML reports...
2022-05-18T04:13:36.3259968Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20220518041335.xml
2022-05-18T04:13:37.1111616Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:13:37.1120839Z 
2022-05-18T04:13:37.1120928Z Running tests...
2022-05-18T04:13:37.1121357Z ----------------------------------------------------------------------
2022-05-18T04:13:37.4023623Z   test_broadcast_coalesced_gloo_cuda (__main__.CommTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3829
2022-05-18T04:13:37.4044752Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3830
2022-05-18T04:13:37.9805292Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:13:38.0125097Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:13:38.2071207Z skip: Need at least 2 CUDA devices (1.095s)
2022-05-18T04:13:38.2071502Z 
2022-05-18T04:13:38.2072030Z ----------------------------------------------------------------------
2022-05-18T04:13:38.2072298Z Ran 1 test in 1.095s
2022-05-18T04:13:38.2072416Z 
2022-05-18T04:13:38.2072495Z OK (skipped=1)
2022-05-18T04:13:38.2072608Z 
2022-05-18T04:13:38.2072696Z Generating XML reports...
2022-05-18T04:13:38.2106115Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20220518041337.xml
2022-05-18T04:13:38.9890341Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:13:38.9899209Z 
2022-05-18T04:13:38.9899309Z Running tests...
2022-05-18T04:13:38.9900314Z ----------------------------------------------------------------------
2022-05-18T04:13:39.2749388Z   test_gloo_barrier_device_ids (__main__.CommTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3862
2022-05-18T04:13:39.2771023Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3863
2022-05-18T04:13:39.8488956Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:13:39.8515278Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:13:39.8697469Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:13:39.8697882Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:13:39.8698954Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:13:39.8699924Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-05-18T04:13:40.0796651Z ok (1.089s)
2022-05-18T04:13:40.0796912Z 
2022-05-18T04:13:40.0797417Z ----------------------------------------------------------------------
2022-05-18T04:13:40.0797729Z Ran 1 test in 1.090s
2022-05-18T04:13:40.0797847Z 
2022-05-18T04:13:40.0797915Z OK
2022-05-18T04:13:40.0798011Z 
2022-05-18T04:13:40.0798105Z Generating XML reports...
2022-05-18T04:13:40.0832754Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20220518041338.xml
2022-05-18T04:13:40.8597928Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:13:40.8607072Z 
2022-05-18T04:13:40.8607213Z Running tests...
2022-05-18T04:13:40.8607603Z ----------------------------------------------------------------------
2022-05-18T04:13:41.1456840Z   test_gloo_warn_not_in_group (__main__.CommTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3901
2022-05-18T04:13:41.1478213Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3902
2022-05-18T04:13:41.7204032Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:13:41.7208878Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:13:41.8503276Z skip: Need at least 2 CUDA devices (0.989s)
2022-05-18T04:13:41.8503494Z 
2022-05-18T04:13:41.8503803Z ----------------------------------------------------------------------
2022-05-18T04:13:41.8504069Z Ran 1 test in 0.989s
2022-05-18T04:13:41.8504186Z 
2022-05-18T04:13:41.8504253Z OK (skipped=1)
2022-05-18T04:13:41.8504394Z 
2022-05-18T04:13:41.8504501Z Generating XML reports...
2022-05-18T04:13:41.8537483Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20220518041340.xml
2022-05-18T04:13:42.6367211Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:13:42.6376406Z 
2022-05-18T04:13:42.6376547Z Running tests...
2022-05-18T04:13:42.6377173Z ----------------------------------------------------------------------
2022-05-18T04:13:42.9237656Z   test_sequence_num_incremented_gloo_default (__main__.CommTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3934
2022-05-18T04:13:42.9258704Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3935
2022-05-18T04:13:43.4994967Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:13:43.5012647Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:13:43.6283514Z skip: Need at least 2 CUDA devices (0.990s)
2022-05-18T04:13:43.6283806Z 
2022-05-18T04:13:43.6284325Z ----------------------------------------------------------------------
2022-05-18T04:13:43.6284602Z Ran 1 test in 0.991s
2022-05-18T04:13:43.6284741Z 
2022-05-18T04:13:43.6284816Z OK (skipped=1)
2022-05-18T04:13:43.6284987Z 
2022-05-18T04:13:43.6285078Z Generating XML reports...
2022-05-18T04:13:43.6317743Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20220518041342.xml
2022-05-18T04:13:44.4056622Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:13:44.4065559Z 
2022-05-18T04:13:44.4065689Z Running tests...
2022-05-18T04:13:44.4066075Z ----------------------------------------------------------------------
2022-05-18T04:13:44.6924068Z   test_sequence_num_incremented_gloo_subgroup (__main__.CommTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3967
2022-05-18T04:13:44.6945078Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3968
2022-05-18T04:13:45.2662092Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:13:45.2678082Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:13:45.3969509Z skip: Need at least 4 CUDA devices (0.990s)
2022-05-18T04:13:45.3969844Z 
2022-05-18T04:13:45.3970273Z ----------------------------------------------------------------------
2022-05-18T04:13:45.3970542Z Ran 1 test in 0.990s
2022-05-18T04:13:45.3970655Z 
2022-05-18T04:13:45.3970729Z OK (skipped=1)
2022-05-18T04:13:45.3970824Z 
2022-05-18T04:13:45.3970909Z Generating XML reports...
2022-05-18T04:13:45.4003577Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20220518041344.xml
2022-05-18T04:13:46.1819373Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:13:46.1829352Z 
2022-05-18T04:13:46.1829600Z Running tests...
2022-05-18T04:13:46.1830245Z ----------------------------------------------------------------------
2022-05-18T04:13:46.4689814Z   test_sequence_num_set_default_pg_gloo (__main__.CommTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4000
2022-05-18T04:13:46.4711838Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4001
2022-05-18T04:13:47.0478719Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:13:47.0479128Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:13:47.1735556Z skip: Need at least 2 CUDA devices (0.990s)
2022-05-18T04:13:47.1735864Z 
2022-05-18T04:13:47.1736267Z ----------------------------------------------------------------------
2022-05-18T04:13:47.1736690Z Ran 1 test in 0.991s
2022-05-18T04:13:47.1736879Z 
2022-05-18T04:13:47.1737012Z OK (skipped=1)
2022-05-18T04:13:47.1737205Z 
2022-05-18T04:13:47.1737309Z Generating XML reports...
2022-05-18T04:13:47.1770817Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20220518041346.xml
2022-05-18T04:13:47.9579780Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:13:47.9589161Z 
2022-05-18T04:13:47.9589243Z Running tests...
2022-05-18T04:13:47.9589968Z ----------------------------------------------------------------------
2022-05-18T04:13:48.2443302Z   test_sequence_num_set_gloo_new_group (__main__.CommTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4033
2022-05-18T04:13:48.2464321Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4034
2022-05-18T04:13:48.8283484Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:13:48.8461932Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:13:49.0489953Z skip: Need at least 2 CUDA devices (1.090s)
2022-05-18T04:13:49.0490166Z 
2022-05-18T04:13:49.0490620Z ----------------------------------------------------------------------
2022-05-18T04:13:49.0490888Z Ran 1 test in 1.090s
2022-05-18T04:13:49.0491004Z 
2022-05-18T04:13:49.0491093Z OK (skipped=1)
2022-05-18T04:13:49.0491229Z 
2022-05-18T04:13:49.0491319Z Generating XML reports...
2022-05-18T04:13:49.0526600Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20220518041347.xml
2022-05-18T04:13:49.8330174Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:13:49.8338524Z 
2022-05-18T04:13:49.8338625Z Running tests...
2022-05-18T04:13:49.8339191Z ----------------------------------------------------------------------
2022-05-18T04:13:49.8345487Z   test_ddp_checkpointing_dynamic_module (__main__.DistributedDataParallelTest)
2022-05-18T04:13:50.1245454Z Dynamic module can be checkpointed, multiple times, with non-reentrant ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4066
2022-05-18T04:13:50.1266964Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4067
2022-05-18T04:13:50.7117442Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:13:50.7126627Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:13:50.8291160Z skip: Need at least 2 CUDA devices (0.995s)
2022-05-18T04:13:50.8291416Z 
2022-05-18T04:13:50.8291882Z ----------------------------------------------------------------------
2022-05-18T04:13:50.8292272Z Ran 1 test in 0.995s
2022-05-18T04:13:50.8292439Z 
2022-05-18T04:13:50.8292556Z OK (skipped=1)
2022-05-18T04:13:50.8292724Z 
2022-05-18T04:13:50.8292850Z Generating XML reports...
2022-05-18T04:13:50.8327664Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220518041349.xml
2022-05-18T04:13:51.6134536Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:13:51.6143392Z 
2022-05-18T04:13:51.6143488Z Running tests...
2022-05-18T04:13:51.6144684Z ----------------------------------------------------------------------
2022-05-18T04:13:51.6150738Z   test_ddp_checkpointing_dynamic_weight_sharing (__main__.DistributedDataParallelTest)
2022-05-18T04:13:51.8998754Z Dynamic module can be checkpointed multiple times with weight sharing ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4099
2022-05-18T04:13:51.9020433Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4100
2022-05-18T04:13:52.4829861Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:13:52.4954297Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:13:52.7046806Z skip: Need at least 2 CUDA devices (1.090s)
2022-05-18T04:13:52.7047102Z 
2022-05-18T04:13:52.7047556Z ----------------------------------------------------------------------
2022-05-18T04:13:52.7047813Z Ran 1 test in 1.090s
2022-05-18T04:13:52.7047928Z 
2022-05-18T04:13:52.7048001Z OK (skipped=1)
2022-05-18T04:13:52.7048112Z 
2022-05-18T04:13:52.7048198Z Generating XML reports...
2022-05-18T04:13:52.7081213Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220518041351.xml
2022-05-18T04:13:53.4875191Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:13:53.4885419Z 
2022-05-18T04:13:53.4885916Z Running tests...
2022-05-18T04:13:53.4886353Z ----------------------------------------------------------------------
2022-05-18T04:13:53.4894534Z   test_ddp_checkpointing_once_use_reentrant_False (__main__.DistributedDataParallelTest)
2022-05-18T04:13:53.7747909Z DDP works as expected when layer is checkpointed only once. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4132
2022-05-18T04:13:53.7769571Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4133
2022-05-18T04:13:54.3520265Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:13:54.3739946Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:13:54.5795388Z skip: Need at least 2 CUDA devices (1.091s)
2022-05-18T04:13:54.5795722Z 
2022-05-18T04:13:54.5796130Z ----------------------------------------------------------------------
2022-05-18T04:13:54.5796367Z Ran 1 test in 1.091s
2022-05-18T04:13:54.5796481Z 
2022-05-18T04:13:54.5796555Z OK (skipped=1)
2022-05-18T04:13:54.5796660Z 
2022-05-18T04:13:54.5796746Z Generating XML reports...
2022-05-18T04:13:54.5832299Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220518041353.xml
2022-05-18T04:13:55.3641862Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:13:55.3651274Z 
2022-05-18T04:13:55.3651421Z Running tests...
2022-05-18T04:13:55.3651957Z ----------------------------------------------------------------------
2022-05-18T04:13:55.3660500Z   test_ddp_checkpointing_once_use_reentrant_True (__main__.DistributedDataParallelTest)
2022-05-18T04:13:55.6529703Z DDP works as expected when layer is checkpointed only once. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4165
2022-05-18T04:13:55.6552062Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4166
2022-05-18T04:13:56.2301123Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:13:56.2318674Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:13:56.3575440Z skip: Need at least 2 CUDA devices (0.992s)
2022-05-18T04:13:56.3575841Z 
2022-05-18T04:13:56.3576720Z ----------------------------------------------------------------------
2022-05-18T04:13:56.3577124Z Ran 1 test in 0.992s
2022-05-18T04:13:56.3577243Z 
2022-05-18T04:13:56.3577322Z OK (skipped=1)
2022-05-18T04:13:56.3577440Z 
2022-05-18T04:13:56.3577534Z Generating XML reports...
2022-05-18T04:13:56.3611815Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220518041355.xml
2022-05-18T04:13:57.1569771Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:13:57.1579502Z 
2022-05-18T04:13:57.1579595Z Running tests...
2022-05-18T04:13:57.1580374Z ----------------------------------------------------------------------
2022-05-18T04:13:57.1587973Z   test_ddp_checkpointing_twice_static_graph_use_reentrant_False (__main__.DistributedDataParallelTest)
2022-05-18T04:13:57.4533664Z Regardless of reentrant or non-reentrant checkpointing impl, ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4198
2022-05-18T04:13:57.4555405Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4199
2022-05-18T04:13:58.0307650Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:13:58.0308140Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:13:58.1579393Z skip: Need at least 2 CUDA devices (1.000s)
2022-05-18T04:13:58.1579679Z 
2022-05-18T04:13:58.1580195Z ----------------------------------------------------------------------
2022-05-18T04:13:58.1580454Z Ran 1 test in 1.000s
2022-05-18T04:13:58.1580569Z 
2022-05-18T04:13:58.1580630Z OK (skipped=1)
2022-05-18T04:13:58.1580737Z 
2022-05-18T04:13:58.1580821Z Generating XML reports...
2022-05-18T04:13:58.1614385Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220518041357.xml
2022-05-18T04:13:58.9409727Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:13:58.9418840Z 
2022-05-18T04:13:58.9418976Z Running tests...
2022-05-18T04:13:58.9419315Z ----------------------------------------------------------------------
2022-05-18T04:13:58.9426381Z   test_ddp_checkpointing_twice_static_graph_use_reentrant_True (__main__.DistributedDataParallelTest)
2022-05-18T04:13:59.2336012Z Regardless of reentrant or non-reentrant checkpointing impl, ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4231
2022-05-18T04:13:59.2358678Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4232
2022-05-18T04:13:59.8149648Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:13:59.8368063Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:14:00.0383675Z skip: Need at least 2 CUDA devices (1.096s)
2022-05-18T04:14:00.0383897Z 
2022-05-18T04:14:00.0384440Z ----------------------------------------------------------------------
2022-05-18T04:14:00.0385041Z Ran 1 test in 1.096s
2022-05-18T04:14:00.0385157Z 
2022-05-18T04:14:00.0385231Z OK (skipped=1)
2022-05-18T04:14:00.0385337Z 
2022-05-18T04:14:00.0385412Z Generating XML reports...
2022-05-18T04:14:00.0418785Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220518041358.xml
2022-05-18T04:14:00.8228103Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:14:00.8236445Z 
2022-05-18T04:14:00.8236546Z Running tests...
2022-05-18T04:14:00.8236974Z ----------------------------------------------------------------------
2022-05-18T04:14:00.8246476Z   test_ddp_checkpointing_twice_use_reentrant_False (__main__.DistributedDataParallelTest)
2022-05-18T04:14:01.1113270Z Checkpoitning twice fails for non-static graph with reentrant checkpoint ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4264
2022-05-18T04:14:01.1134898Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4265
2022-05-18T04:14:01.6908546Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:14:01.6934137Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:14:01.8158116Z skip: Need at least 2 CUDA devices (0.992s)
2022-05-18T04:14:01.8158545Z 
2022-05-18T04:14:01.8158932Z ----------------------------------------------------------------------
2022-05-18T04:14:01.8159169Z Ran 1 test in 0.992s
2022-05-18T04:14:01.8159280Z 
2022-05-18T04:14:01.8159352Z OK (skipped=1)
2022-05-18T04:14:01.8159460Z 
2022-05-18T04:14:01.8159546Z Generating XML reports...
2022-05-18T04:14:01.8193465Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220518041400.xml
2022-05-18T04:14:02.6021931Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:14:02.6031452Z 
2022-05-18T04:14:02.6031586Z Running tests...
2022-05-18T04:14:02.6032202Z ----------------------------------------------------------------------
2022-05-18T04:14:02.6041085Z   test_ddp_checkpointing_twice_use_reentrant_True (__main__.DistributedDataParallelTest)
2022-05-18T04:14:02.8910132Z Checkpoitning twice fails for non-static graph with reentrant checkpoint ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4297
2022-05-18T04:14:02.8931404Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4298
2022-05-18T04:14:03.4680028Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:14:03.4689989Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:14:03.5955159Z skip: Need at least 2 CUDA devices (0.992s)
2022-05-18T04:14:03.5955439Z 
2022-05-18T04:14:03.5955810Z ----------------------------------------------------------------------
2022-05-18T04:14:03.5956057Z Ran 1 test in 0.992s
2022-05-18T04:14:03.5956174Z 
2022-05-18T04:14:03.5956281Z OK (skipped=1)
2022-05-18T04:14:03.5956413Z 
2022-05-18T04:14:03.5956498Z Generating XML reports...
2022-05-18T04:14:03.5991456Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220518041402.xml
2022-05-18T04:14:04.3774791Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:14:04.3784291Z 
2022-05-18T04:14:04.3784396Z Running tests...
2022-05-18T04:14:04.3784805Z ----------------------------------------------------------------------
2022-05-18T04:14:04.3791566Z   test_ddp_checkpointing_twice_weight_sharing (__main__.DistributedDataParallelTest)
2022-05-18T04:14:04.6666121Z Checkpointing should work with static graph in the case of checkpointing ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4330
2022-05-18T04:14:04.6687132Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4331
2022-05-18T04:14:05.2420915Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:14:05.2447783Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:14:05.3708817Z skip: Need at least 2 CUDA devices (0.992s)
2022-05-18T04:14:05.3709187Z 
2022-05-18T04:14:05.3709693Z ----------------------------------------------------------------------
2022-05-18T04:14:05.3710139Z Ran 1 test in 0.992s
2022-05-18T04:14:05.3710341Z 
2022-05-18T04:14:05.3710476Z OK (skipped=1)
2022-05-18T04:14:05.3710640Z 
2022-05-18T04:14:05.3710753Z Generating XML reports...
2022-05-18T04:14:05.3743698Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220518041404.xml
2022-05-18T04:14:06.1607858Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:14:06.1616533Z 
2022-05-18T04:14:06.1616666Z Running tests...
2022-05-18T04:14:06.1617246Z ----------------------------------------------------------------------
2022-05-18T04:14:06.1626472Z   test_ddp_checkpointing_unused_params_use_reentrant_False (__main__.DistributedDataParallelTest)
2022-05-18T04:14:06.4493243Z With reentrant autograd checkpointing impl, DDP will fail when there are ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4363
2022-05-18T04:14:06.4514337Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4364
2022-05-18T04:14:07.0278593Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:14:07.0560838Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:14:07.2541338Z skip: Need at least 2 CUDA devices (1.092s)
2022-05-18T04:14:07.2541530Z 
2022-05-18T04:14:07.2541913Z ----------------------------------------------------------------------
2022-05-18T04:14:07.2542184Z Ran 1 test in 1.092s
2022-05-18T04:14:07.2542285Z 
2022-05-18T04:14:07.2542377Z OK (skipped=1)
2022-05-18T04:14:07.2542482Z 
2022-05-18T04:14:07.2542604Z Generating XML reports...
2022-05-18T04:14:07.2576430Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220518041406.xml
2022-05-18T04:14:08.0429740Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:14:08.0438299Z 
2022-05-18T04:14:08.0438561Z Running tests...
2022-05-18T04:14:08.0439024Z ----------------------------------------------------------------------
2022-05-18T04:14:08.0447930Z   test_ddp_checkpointing_unused_params_use_reentrant_True (__main__.DistributedDataParallelTest)
2022-05-18T04:14:08.3314547Z With reentrant autograd checkpointing impl, DDP will fail when there are ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4396
2022-05-18T04:14:08.3335544Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4397
2022-05-18T04:14:08.9128252Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:14:08.9128970Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:14:09.0358943Z skip: Need at least 2 CUDA devices (0.992s)
2022-05-18T04:14:09.0359262Z 
2022-05-18T04:14:09.0359599Z ----------------------------------------------------------------------
2022-05-18T04:14:09.0359848Z Ran 1 test in 0.992s
2022-05-18T04:14:09.0359966Z 
2022-05-18T04:14:09.0360040Z OK (skipped=1)
2022-05-18T04:14:09.0360149Z 
2022-05-18T04:14:09.0360232Z Generating XML reports...
2022-05-18T04:14:09.0393621Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220518041408.xml
2022-05-18T04:14:09.8202307Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:14:09.8210758Z 
2022-05-18T04:14:09.8211128Z Running tests...
2022-05-18T04:14:09.8211531Z ----------------------------------------------------------------------
2022-05-18T04:14:09.8221788Z   test_ddp_checkpointing_weight_sharing_use_reentrant_False (__main__.DistributedDataParallelTest)
2022-05-18T04:14:10.1079958Z Test that checkpointing with weight sharing works. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4429
2022-05-18T04:14:10.1101337Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4430
2022-05-18T04:14:10.7074452Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:14:10.7076627Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:14:10.9127204Z skip: Need at least 2 CUDA devices (1.091s)
2022-05-18T04:14:10.9127518Z 
2022-05-18T04:14:10.9128035Z ----------------------------------------------------------------------
2022-05-18T04:14:10.9128297Z Ran 1 test in 1.092s
2022-05-18T04:14:10.9128410Z 
2022-05-18T04:14:10.9128485Z OK (skipped=1)
2022-05-18T04:14:10.9128614Z 
2022-05-18T04:14:10.9128686Z Generating XML reports...
2022-05-18T04:14:10.9165277Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220518041409.xml
2022-05-18T04:14:11.6941176Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:14:11.6950547Z 
2022-05-18T04:14:11.6950644Z Running tests...
2022-05-18T04:14:11.6951387Z ----------------------------------------------------------------------
2022-05-18T04:14:11.6961635Z   test_ddp_checkpointing_weight_sharing_use_reentrant_True (__main__.DistributedDataParallelTest)
2022-05-18T04:14:11.9804119Z Test that checkpointing with weight sharing works. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4462
2022-05-18T04:14:11.9826014Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4463
2022-05-18T04:14:12.5550136Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:14:12.5628568Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:14:12.7852111Z skip: Need at least 2 CUDA devices (1.090s)
2022-05-18T04:14:12.7852445Z 
2022-05-18T04:14:12.7852956Z ----------------------------------------------------------------------
2022-05-18T04:14:12.7853365Z Ran 1 test in 1.090s
2022-05-18T04:14:12.7853575Z 
2022-05-18T04:14:12.7853709Z OK (skipped=1)
2022-05-18T04:14:12.7853882Z 
2022-05-18T04:14:12.7853972Z Generating XML reports...
2022-05-18T04:14:12.7887950Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220518041411.xml
2022-05-18T04:14:13.5705409Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:14:13.5714575Z 
2022-05-18T04:14:13.5715220Z Running tests...
2022-05-18T04:14:13.5715618Z ----------------------------------------------------------------------
2022-05-18T04:14:13.5722535Z   test_ddp_comm_hook_future_passing_cpu (__main__.DistributedDataParallelTest)
2022-05-18T04:14:13.8574606Z This unit test verifies whether the Future object is passed properly. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4495
2022-05-18T04:14:13.8596345Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4496
2022-05-18T04:14:14.4326610Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:14:14.4331685Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:14:14.4505994Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmaxm9_eb
2022-05-18T04:14:14.4506629Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp88faijke
2022-05-18T04:14:14.4507875Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp88faijke/_remote_module_non_scriptable.py
2022-05-18T04:14:14.4508565Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmaxm9_eb/_remote_module_non_scriptable.py
2022-05-18T04:14:14.6621785Z ok (1.090s)
2022-05-18T04:14:14.6622039Z 
2022-05-18T04:14:14.6623057Z ----------------------------------------------------------------------
2022-05-18T04:14:14.6623395Z Ran 1 test in 1.091s
2022-05-18T04:14:14.6623499Z 
2022-05-18T04:14:14.6623560Z OK
2022-05-18T04:14:14.6623651Z 
2022-05-18T04:14:14.6623749Z Generating XML reports...
2022-05-18T04:14:14.6657142Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220518041413.xml
2022-05-18T04:14:15.4421422Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:14:15.4431031Z 
2022-05-18T04:14:15.4431132Z Running tests...
2022-05-18T04:14:15.4431701Z ----------------------------------------------------------------------
2022-05-18T04:14:15.4438580Z   test_ddp_comm_hook_future_passing_gpu_gloo (__main__.DistributedDataParallelTest)
2022-05-18T04:14:15.7308164Z This unit test verifies whether the Future object is passed properly using gloo backend. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4534
2022-05-18T04:14:15.7329255Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4535
2022-05-18T04:14:16.3057609Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:14:16.3066838Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:14:16.4352947Z skip: Need at least 2 CUDA devices (0.992s)
2022-05-18T04:14:16.4353259Z 
2022-05-18T04:14:16.4353762Z ----------------------------------------------------------------------
2022-05-18T04:14:16.4354144Z Ran 1 test in 0.992s
2022-05-18T04:14:16.4354259Z 
2022-05-18T04:14:16.4354332Z OK (skipped=1)
2022-05-18T04:14:16.4354427Z 
2022-05-18T04:14:16.4354513Z Generating XML reports...
2022-05-18T04:14:16.4388582Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220518041415.xml
2022-05-18T04:14:17.2213933Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:14:17.2223161Z 
2022-05-18T04:14:17.2223687Z Running tests...
2022-05-18T04:14:17.2224157Z ----------------------------------------------------------------------
2022-05-18T04:14:17.2232038Z   test_ddp_comm_hook_register_just_once (__main__.DistributedDataParallelTest)
2022-05-18T04:14:17.5114918Z DDP communication hook can only be registered once. This test validates whether ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4567
2022-05-18T04:14:17.5137594Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4568
2022-05-18T04:14:18.0867236Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:14:18.0875652Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:14:18.1148671Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj9wimyqk
2022-05-18T04:14:18.1149223Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpejmdsbq5
2022-05-18T04:14:18.1150320Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj9wimyqk/_remote_module_non_scriptable.py
2022-05-18T04:14:18.1150979Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpejmdsbq5/_remote_module_non_scriptable.py
2022-05-18T04:14:18.3162551Z ok (1.094s)
2022-05-18T04:14:18.3162782Z 
2022-05-18T04:14:18.3163302Z ----------------------------------------------------------------------
2022-05-18T04:14:18.3163704Z Ran 1 test in 1.094s
2022-05-18T04:14:18.3163820Z 
2022-05-18T04:14:18.3163881Z OK
2022-05-18T04:14:18.3163972Z 
2022-05-18T04:14:18.3164066Z Generating XML reports...
2022-05-18T04:14:18.3198149Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220518041417.xml
2022-05-18T04:14:19.0962817Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:14:19.0972040Z 
2022-05-18T04:14:19.0972181Z Running tests...
2022-05-18T04:14:19.0972822Z ----------------------------------------------------------------------
2022-05-18T04:14:19.0982840Z   test_ddp_comm_hook_sparse_gradients (__main__.DistributedDataParallelTest)
2022-05-18T04:14:19.3858192Z Runs "test_sparse_gradients" unit test with DDP communication hook. We define a ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4606
2022-05-18T04:14:19.3880362Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4607
2022-05-18T04:14:19.9703338Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:14:19.9703783Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:14:19.9983069Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvn1iduou
2022-05-18T04:14:19.9983683Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_l0i4b5l
2022-05-18T04:14:19.9984411Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvn1iduou/_remote_module_non_scriptable.py
2022-05-18T04:14:19.9985480Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_l0i4b5l/_remote_module_non_scriptable.py
2022-05-18T04:14:20.1905008Z ok (1.093s)
2022-05-18T04:14:20.1905490Z 
2022-05-18T04:14:20.1906067Z ----------------------------------------------------------------------
2022-05-18T04:14:20.1906356Z Ran 1 test in 1.093s
2022-05-18T04:14:20.1906481Z 
2022-05-18T04:14:20.1906529Z OK
2022-05-18T04:14:20.1906624Z 
2022-05-18T04:14:20.1906715Z Generating XML reports...
2022-05-18T04:14:20.1940453Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220518041419.xml
2022-05-18T04:14:20.9628036Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:14:20.9636685Z 
2022-05-18T04:14:20.9636805Z Running tests...
2022-05-18T04:14:20.9637407Z ----------------------------------------------------------------------
2022-05-18T04:14:20.9646434Z   test_ddp_invalid_comm_hook_init (__main__.DistributedDataParallelTest)
2022-05-18T04:14:21.2515156Z This unit test makes sure that register_comm_hook properly checks the format ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4651
2022-05-18T04:14:21.2536111Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4652
2022-05-18T04:14:21.8267069Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:14:21.8270087Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:14:21.8544672Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpazy4dhxg
2022-05-18T04:14:21.8545658Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpatq0emmf
2022-05-18T04:14:21.8546320Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpazy4dhxg/_remote_module_non_scriptable.py
2022-05-18T04:14:21.8547021Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpatq0emmf/_remote_module_non_scriptable.py
2022-05-18T04:14:22.0561585Z ok (1.092s)
2022-05-18T04:14:22.0561816Z 
2022-05-18T04:14:22.0562352Z ----------------------------------------------------------------------
2022-05-18T04:14:22.0562665Z Ran 1 test in 1.092s
2022-05-18T04:14:22.0562780Z 
2022-05-18T04:14:22.0562842Z OK
2022-05-18T04:14:22.0562934Z 
2022-05-18T04:14:22.0563026Z Generating XML reports...
2022-05-18T04:14:22.0598119Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220518041420.xml
2022-05-18T04:14:22.8361810Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:14:22.8370612Z 
2022-05-18T04:14:22.8371037Z Running tests...
2022-05-18T04:14:22.8371409Z ----------------------------------------------------------------------
2022-05-18T04:14:22.8382333Z   test_ddp_invalid_comm_hook_return_type (__main__.DistributedDataParallelTest)
2022-05-18T04:14:23.1221753Z This test checks whether return annotation checked properly if defined. It also ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4690
2022-05-18T04:14:23.1243531Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4691
2022-05-18T04:14:23.7011449Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:14:23.7258780Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:14:23.7485516Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppbkd1bhf
2022-05-18T04:14:23.7486141Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0_fomfqf
2022-05-18T04:14:23.7486895Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppbkd1bhf/_remote_module_non_scriptable.py
2022-05-18T04:14:23.7488236Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0_fomfqf/_remote_module_non_scriptable.py
2022-05-18T04:14:23.9268933Z ok (1.090s)
2022-05-18T04:14:23.9269163Z 
2022-05-18T04:14:23.9269629Z ----------------------------------------------------------------------
2022-05-18T04:14:23.9270027Z Ran 1 test in 1.090s
2022-05-18T04:14:23.9270189Z 
2022-05-18T04:14:23.9270283Z OK
2022-05-18T04:14:23.9270431Z 
2022-05-18T04:14:23.9270584Z Generating XML reports...
2022-05-18T04:14:23.9304501Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220518041422.xml
2022-05-18T04:14:24.6988944Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:14:24.6997575Z 
2022-05-18T04:14:24.6997678Z Running tests...
2022-05-18T04:14:24.6998257Z ----------------------------------------------------------------------
2022-05-18T04:14:24.7013577Z   test_find_unused_parameters_when_unused_parameters_empty (__main__.DistributedDataParallelTest)
2022-05-18T04:14:24.9849542Z An empty unused_parameters array does not imply find_unused_parameters = ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4729
2022-05-18T04:14:24.9871673Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4730
2022-05-18T04:14:25.5640430Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:14:25.5648838Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:14:25.6895816Z skip: Need at least 2 CUDA devices (0.989s)
2022-05-18T04:14:25.6896074Z 
2022-05-18T04:14:25.6896531Z ----------------------------------------------------------------------
2022-05-18T04:14:25.6896905Z Ran 1 test in 0.990s
2022-05-18T04:14:25.6897076Z 
2022-05-18T04:14:25.6897197Z OK (skipped=1)
2022-05-18T04:14:25.6897368Z 
2022-05-18T04:14:25.6897508Z Generating XML reports...
2022-05-18T04:14:25.6931780Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220518041424.xml
2022-05-18T04:14:26.4725145Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:14:26.4734652Z 
2022-05-18T04:14:26.4734746Z Running tests...
2022-05-18T04:14:26.4735648Z ----------------------------------------------------------------------
2022-05-18T04:14:26.7587422Z   test_global_local_unused_params_grad (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4762
2022-05-18T04:14:26.7608984Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4763
2022-05-18T04:14:27.3320046Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:14:27.3339770Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:14:27.4633438Z skip: Need at least 2 CUDA devices (0.990s)
2022-05-18T04:14:27.4633647Z 
2022-05-18T04:14:27.4634124Z ----------------------------------------------------------------------
2022-05-18T04:14:27.4634448Z Ran 1 test in 0.990s
2022-05-18T04:14:27.4634563Z 
2022-05-18T04:14:27.4634637Z OK (skipped=1)
2022-05-18T04:14:27.4634731Z 
2022-05-18T04:14:27.4634817Z Generating XML reports...
2022-05-18T04:14:27.4667729Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220518041426.xml
2022-05-18T04:14:28.2575325Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:14:28.2584564Z 
2022-05-18T04:14:28.2584668Z Running tests...
2022-05-18T04:14:28.2585123Z ----------------------------------------------------------------------
2022-05-18T04:14:28.5386926Z   test_global_local_unused_params_grad_with_grad_is_view (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4795
2022-05-18T04:14:28.5408634Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4796
2022-05-18T04:14:29.1164846Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:14:29.1181190Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:14:29.2432275Z skip: Need at least 2 CUDA devices (0.984s)
2022-05-18T04:14:29.2432589Z 
2022-05-18T04:14:29.2432939Z ----------------------------------------------------------------------
2022-05-18T04:14:29.2433193Z Ran 1 test in 0.985s
2022-05-18T04:14:29.2433308Z 
2022-05-18T04:14:29.2433382Z OK (skipped=1)
2022-05-18T04:14:29.2433488Z 
2022-05-18T04:14:29.2433579Z Generating XML reports...
2022-05-18T04:14:29.2470451Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220518041428.xml
2022-05-18T04:14:30.0113314Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:14:30.0123028Z 
2022-05-18T04:14:30.0123460Z Running tests...
2022-05-18T04:14:30.0123906Z ----------------------------------------------------------------------
2022-05-18T04:14:30.2973493Z   test_global_local_unused_params_grad_with_static_graph (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4828
2022-05-18T04:14:30.2994797Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4829
2022-05-18T04:14:30.8654648Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:14:30.8661821Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:14:31.0018846Z skip: Need at least 2 CUDA devices (0.989s)
2022-05-18T04:14:31.0019134Z 
2022-05-18T04:14:31.0019639Z ----------------------------------------------------------------------
2022-05-18T04:14:31.0020020Z Ran 1 test in 0.990s
2022-05-18T04:14:31.0020134Z 
2022-05-18T04:14:31.0020205Z OK (skipped=1)
2022-05-18T04:14:31.0020317Z 
2022-05-18T04:14:31.0020409Z Generating XML reports...
2022-05-18T04:14:31.0054327Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220518041430.xml
2022-05-18T04:14:31.7602729Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:14:31.7611493Z 
2022-05-18T04:14:31.7611585Z Running tests...
2022-05-18T04:14:31.7612010Z ----------------------------------------------------------------------
2022-05-18T04:14:32.0398553Z   test_gloo_backend_1gpu_module_device_ids_integer_list (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4861
2022-05-18T04:14:32.0419832Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4862
2022-05-18T04:14:32.6217739Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:14:32.6578211Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:14:32.8443928Z skip: Need at least 2 CUDA devices (1.083s)
2022-05-18T04:14:32.8444238Z 
2022-05-18T04:14:32.8444811Z ----------------------------------------------------------------------
2022-05-18T04:14:32.8445078Z Ran 1 test in 1.083s
2022-05-18T04:14:32.8445192Z 
2022-05-18T04:14:32.8445266Z OK (skipped=1)
2022-05-18T04:14:32.8445374Z 
2022-05-18T04:14:32.8445464Z Generating XML reports...
2022-05-18T04:14:32.8480856Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220518041431.xml
2022-05-18T04:14:33.6077425Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:14:33.6087134Z 
2022-05-18T04:14:33.6087436Z Running tests...
2022-05-18T04:14:33.6088065Z ----------------------------------------------------------------------
2022-05-18T04:14:33.8881886Z   test_gloo_backend_1gpu_module_device_ids_torch_device_list (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4894
2022-05-18T04:14:33.8903348Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4895
2022-05-18T04:14:34.4543217Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:14:34.4546685Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:14:34.5926364Z skip: Need at least 2 CUDA devices (0.984s)
2022-05-18T04:14:34.5926802Z 
2022-05-18T04:14:34.5927278Z ----------------------------------------------------------------------
2022-05-18T04:14:34.5927555Z Ran 1 test in 0.984s
2022-05-18T04:14:34.5927682Z 
2022-05-18T04:14:34.5927746Z OK (skipped=1)
2022-05-18T04:14:34.5927856Z 
2022-05-18T04:14:34.5927939Z Generating XML reports...
2022-05-18T04:14:34.5962343Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220518041433.xml
2022-05-18T04:14:35.3440251Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:14:35.3449912Z 
2022-05-18T04:14:35.3450194Z Running tests...
2022-05-18T04:14:35.3450808Z ----------------------------------------------------------------------
2022-05-18T04:14:35.6284886Z   test_gloo_backend_2gpu_module (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4927
2022-05-18T04:14:35.6307896Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4928
2022-05-18T04:14:36.1992157Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:14:36.2365805Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:14:36.4333012Z skip: Need at least 4 CUDA devices (1.088s)
2022-05-18T04:14:36.4333188Z 
2022-05-18T04:14:36.4333599Z ----------------------------------------------------------------------
2022-05-18T04:14:36.4333872Z Ran 1 test in 1.088s
2022-05-18T04:14:36.4333987Z 
2022-05-18T04:14:36.4334046Z OK (skipped=1)
2022-05-18T04:14:36.4334154Z 
2022-05-18T04:14:36.4334238Z Generating XML reports...
2022-05-18T04:14:36.4369596Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220518041435.xml
2022-05-18T04:14:37.2040510Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:14:37.2049154Z 
2022-05-18T04:14:37.2049709Z Running tests...
2022-05-18T04:14:37.2050118Z ----------------------------------------------------------------------
2022-05-18T04:14:37.4939126Z   test_gloo_backend_4gpu_module (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4960
2022-05-18T04:14:37.4961757Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4961
2022-05-18T04:14:38.0646353Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:14:38.0931501Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:14:38.1985613Z skip: Need at least 8 CUDA devices (0.993s)
2022-05-18T04:14:38.1985903Z 
2022-05-18T04:14:38.1986216Z ----------------------------------------------------------------------
2022-05-18T04:14:38.1986465Z Ran 1 test in 0.994s
2022-05-18T04:14:38.1986578Z 
2022-05-18T04:14:38.1986649Z OK (skipped=1)
2022-05-18T04:14:38.1986743Z 
2022-05-18T04:14:38.1986831Z Generating XML reports...
2022-05-18T04:14:38.2019787Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220518041437.xml
2022-05-18T04:14:38.9527454Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:14:38.9536396Z 
2022-05-18T04:14:38.9536506Z Running tests...
2022-05-18T04:14:38.9537066Z ----------------------------------------------------------------------
2022-05-18T04:14:39.2349488Z   test_gloo_backend_cpu_module (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4993
2022-05-18T04:14:39.2371260Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4994
2022-05-18T04:14:39.8124060Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:14:39.8165436Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:14:39.8399624Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxgi6pdu3
2022-05-18T04:14:39.8400319Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqqd9bmfk
2022-05-18T04:14:39.8401416Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxgi6pdu3/_remote_module_non_scriptable.py
2022-05-18T04:14:39.8401863Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqqd9bmfk/_remote_module_non_scriptable.py
2022-05-18T04:14:39.8550002Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:14:39.8550713Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:14:40.0395424Z ok (1.086s)
2022-05-18T04:14:40.0395636Z 
2022-05-18T04:14:40.0396166Z ----------------------------------------------------------------------
2022-05-18T04:14:40.0396515Z Ran 1 test in 1.086s
2022-05-18T04:14:40.0396630Z 
2022-05-18T04:14:40.0396691Z OK
2022-05-18T04:14:40.0396782Z 
2022-05-18T04:14:40.0396877Z Generating XML reports...
2022-05-18T04:14:40.0429719Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220518041438.xml
2022-05-18T04:14:40.8008916Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:14:40.8017668Z 
2022-05-18T04:14:40.8017748Z Running tests...
2022-05-18T04:14:40.8018273Z ----------------------------------------------------------------------
2022-05-18T04:14:41.0836153Z   test_gloo_backend_cpu_module_grad_is_view (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5032
2022-05-18T04:14:41.0857562Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5033
2022-05-18T04:14:41.6529186Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:14:41.6930389Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:14:41.7104821Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjcikq_9l
2022-05-18T04:14:41.7106888Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjcikq_9l/_remote_module_non_scriptable.py
2022-05-18T04:14:41.7107698Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7l1hghha
2022-05-18T04:14:41.7109334Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7l1hghha/_remote_module_non_scriptable.py
2022-05-18T04:14:41.7253880Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:14:41.7254479Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:14:41.8881506Z ok (1.086s)
2022-05-18T04:14:41.8881776Z 
2022-05-18T04:14:41.8882289Z ----------------------------------------------------------------------
2022-05-18T04:14:41.8882554Z Ran 1 test in 1.086s
2022-05-18T04:14:41.8882669Z 
2022-05-18T04:14:41.8882718Z OK
2022-05-18T04:14:41.8882810Z 
2022-05-18T04:14:41.8882906Z Generating XML reports...
2022-05-18T04:14:41.8918771Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220518041440.xml
2022-05-18T04:14:42.6537721Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:14:42.6547581Z 
2022-05-18T04:14:42.6548149Z Running tests...
2022-05-18T04:14:42.6548768Z ----------------------------------------------------------------------
2022-05-18T04:14:42.6562489Z   test_ignored_output (__main__.DistributedDataParallelTest)
2022-05-18T04:14:42.9350822Z Test that the output of a model can be ignored and that there is no ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5071
2022-05-18T04:14:42.9372021Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5072
2022-05-18T04:14:43.5383511Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:14:43.5404494Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:14:43.5659580Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl54xr8bd
2022-05-18T04:14:43.5660308Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6tdli3ja
2022-05-18T04:14:43.5661440Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl54xr8bd/_remote_module_non_scriptable.py
2022-05-18T04:14:43.5662102Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6tdli3ja/_remote_module_non_scriptable.py
2022-05-18T04:14:43.5829562Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:14:43.5830030Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:14:43.7396559Z ok (1.085s)
2022-05-18T04:14:43.7396756Z 
2022-05-18T04:14:43.7397106Z ----------------------------------------------------------------------
2022-05-18T04:14:43.7397405Z Ran 1 test in 1.085s
2022-05-18T04:14:43.7397522Z 
2022-05-18T04:14:43.7397584Z OK
2022-05-18T04:14:43.7397675Z 
2022-05-18T04:14:43.7397760Z Generating XML reports...
2022-05-18T04:14:43.7431909Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220518041442.xml
2022-05-18T04:14:44.4938658Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:14:44.4947347Z 
2022-05-18T04:14:44.4947483Z Running tests...
2022-05-18T04:14:44.4948063Z ----------------------------------------------------------------------
2022-05-18T04:14:44.4963079Z   test_ignored_output_with_unused_parameters (__main__.DistributedDataParallelTest)
2022-05-18T04:14:44.7743085Z Test that the output of a model can be ignored and that there is no ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5116
2022-05-18T04:14:44.7764845Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5117
2022-05-18T04:14:45.3433569Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:14:45.3444831Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:14:45.3618494Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkygdfhli
2022-05-18T04:14:45.3619128Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz80a8zqw
2022-05-18T04:14:45.3620653Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkygdfhli/_remote_module_non_scriptable.py
2022-05-18T04:14:45.3621164Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz80a8zqw/_remote_module_non_scriptable.py
2022-05-18T04:14:45.5790411Z ok (1.084s)
2022-05-18T04:14:45.5790610Z 
2022-05-18T04:14:45.5791145Z ----------------------------------------------------------------------
2022-05-18T04:14:45.5791544Z Ran 1 test in 1.084s
2022-05-18T04:14:45.5791662Z 
2022-05-18T04:14:45.5791723Z OK
2022-05-18T04:14:45.5791815Z 
2022-05-18T04:14:45.5791895Z Generating XML reports...
2022-05-18T04:14:45.5825059Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220518041444.xml
2022-05-18T04:14:46.3398765Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:14:46.3407936Z 
2022-05-18T04:14:46.3408359Z Running tests...
2022-05-18T04:14:46.3408782Z ----------------------------------------------------------------------
2022-05-18T04:14:46.6210782Z   test_invalid_powerSGD_state (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5161
2022-05-18T04:14:46.6232220Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5162
2022-05-18T04:14:47.2207132Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:14:47.2211830Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-05-18T04:14:47.2213426Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-05-18T04:14:47.2214393Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-05-18T04:14:47.2215194Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-05-18T04:14:47.2216011Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-05-18T04:14:47.2216812Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-05-18T04:14:47.2337796Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:14:47.2342467Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-05-18T04:14:47.2343988Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-05-18T04:14:47.2345017Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-05-18T04:14:47.2345967Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-05-18T04:14:47.2346909Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-05-18T04:14:47.2347868Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-05-18T04:14:47.4256943Z ok (1.085s)
2022-05-18T04:14:47.4257128Z 
2022-05-18T04:14:47.4257491Z ----------------------------------------------------------------------
2022-05-18T04:14:47.4257824Z Ran 1 test in 1.085s
2022-05-18T04:14:47.4257982Z 
2022-05-18T04:14:47.4258044Z OK
2022-05-18T04:14:47.4258136Z 
2022-05-18T04:14:47.4258214Z Generating XML reports...
2022-05-18T04:14:47.4292673Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220518041446.xml
2022-05-18T04:14:48.1841565Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:14:48.1850861Z 
2022-05-18T04:14:48.1850992Z Running tests...
2022-05-18T04:14:48.1851431Z ----------------------------------------------------------------------
2022-05-18T04:14:48.4668904Z   test_save_load_checkpoint (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5194
2022-05-18T04:14:48.4690420Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5195
2022-05-18T04:14:49.0366012Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:14:49.0367005Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:14:49.1713312Z skip: Need at least 2 CUDA devices (0.986s)
2022-05-18T04:14:49.1713494Z 
2022-05-18T04:14:49.1713845Z ----------------------------------------------------------------------
2022-05-18T04:14:49.1714091Z Ran 1 test in 0.986s
2022-05-18T04:14:49.1714205Z 
2022-05-18T04:14:49.1715899Z OK (skipped=1)
2022-05-18T04:14:49.1716125Z 
2022-05-18T04:14:49.1716276Z Generating XML reports...
2022-05-18T04:14:49.1749884Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220518041448.xml
2022-05-18T04:14:49.9418542Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:14:49.9427493Z 
2022-05-18T04:14:49.9427654Z Running tests...
2022-05-18T04:14:49.9428265Z ----------------------------------------------------------------------
2022-05-18T04:14:50.2251937Z   test_sparse_gradients (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5227
2022-05-18T04:14:50.2273750Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5228
2022-05-18T04:14:50.7950967Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:14:50.7951570Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:14:50.8129643Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg2fb9wd3
2022-05-18T04:14:50.8130301Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0z9crx6h
2022-05-18T04:14:50.8131274Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg2fb9wd3/_remote_module_non_scriptable.py
2022-05-18T04:14:50.8131981Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0z9crx6h/_remote_module_non_scriptable.py
2022-05-18T04:14:51.0299055Z ok (1.087s)
2022-05-18T04:14:51.0299781Z 
2022-05-18T04:14:51.0300182Z ----------------------------------------------------------------------
2022-05-18T04:14:51.0300643Z Ran 1 test in 1.087s
2022-05-18T04:14:51.0300854Z 
2022-05-18T04:14:51.0300958Z OK
2022-05-18T04:14:51.0301060Z 
2022-05-18T04:14:51.0301153Z Generating XML reports...
2022-05-18T04:14:51.0335821Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220518041449.xml
2022-05-18T04:14:51.7962712Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:14:51.7971967Z 
2022-05-18T04:14:51.7972088Z Running tests...
2022-05-18T04:14:51.7972655Z ----------------------------------------------------------------------
2022-05-18T04:14:52.0748541Z   test_sparse_gradients_grad_is_view (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5272
2022-05-18T04:14:52.0770669Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5273
2022-05-18T04:14:52.6482209Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:14:52.6498742Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:14:52.6761514Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9njlz698
2022-05-18T04:14:52.6762040Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6_zl43zm
2022-05-18T04:14:52.6763490Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9njlz698/_remote_module_non_scriptable.py
2022-05-18T04:14:52.6764303Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6_zl43zm/_remote_module_non_scriptable.py
2022-05-18T04:14:52.8795633Z ok (1.082s)
2022-05-18T04:14:52.8795885Z 
2022-05-18T04:14:52.8796432Z ----------------------------------------------------------------------
2022-05-18T04:14:52.8796859Z Ran 1 test in 1.082s
2022-05-18T04:14:52.8796986Z 
2022-05-18T04:14:52.8797048Z OK
2022-05-18T04:14:52.8797139Z 
2022-05-18T04:14:52.8797233Z Generating XML reports...
2022-05-18T04:14:52.8831654Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220518041451.xml
2022-05-18T04:14:53.6350477Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:14:53.6358588Z 
2022-05-18T04:14:53.6358705Z Running tests...
2022-05-18T04:14:53.6359132Z ----------------------------------------------------------------------
2022-05-18T04:14:53.9170770Z   test_sync_batch_norm_empty_input (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5317
2022-05-18T04:14:53.9191187Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5318
2022-05-18T04:14:54.5223716Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:14:54.5230646Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:14:54.7216407Z skip: Need at least 2 CUDA devices (1.085s)
2022-05-18T04:14:54.7216732Z 
2022-05-18T04:14:54.7217137Z ----------------------------------------------------------------------
2022-05-18T04:14:54.7217389Z Ran 1 test in 1.086s
2022-05-18T04:14:54.7217492Z 
2022-05-18T04:14:54.7217576Z OK (skipped=1)
2022-05-18T04:14:54.7217683Z 
2022-05-18T04:14:54.7217771Z Generating XML reports...
2022-05-18T04:14:54.7251493Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220518041453.xml
2022-05-18T04:14:55.4775311Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:14:55.4784196Z 
2022-05-18T04:14:55.4784339Z Running tests...
2022-05-18T04:14:55.4784784Z ----------------------------------------------------------------------
2022-05-18T04:14:55.7591503Z   test_sync_batch_norm_only_empty_input (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5350
2022-05-18T04:14:55.7613228Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5351
2022-05-18T04:14:56.3281985Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:14:56.3306408Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:14:56.4636364Z skip: Need at least 2 CUDA devices (0.985s)
2022-05-18T04:14:56.4636684Z 
2022-05-18T04:14:56.4637171Z ----------------------------------------------------------------------
2022-05-18T04:14:56.4637548Z Ran 1 test in 0.985s
2022-05-18T04:14:56.4637674Z 
2022-05-18T04:14:56.4637752Z OK (skipped=1)
2022-05-18T04:14:56.4637860Z 
2022-05-18T04:14:56.4637954Z Generating XML reports...
2022-05-18T04:14:56.4672063Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220518041455.xml
2022-05-18T04:14:57.2171559Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:14:57.2180126Z 
2022-05-18T04:14:57.2180221Z Running tests...
2022-05-18T04:14:57.2180589Z ----------------------------------------------------------------------
2022-05-18T04:14:57.4998449Z   test_allgather_basics (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5383
2022-05-18T04:14:57.5020365Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5384
2022-05-18T04:14:57.5042374Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5385
2022-05-18T04:14:57.5066086Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5386
2022-05-18T04:14:58.1351790Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:14:58.1605354Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:14:58.1618349Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:14:58.1719745Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:14:58.4095945Z ok (1.191s)
2022-05-18T04:14:58.4096104Z 
2022-05-18T04:14:58.4096455Z ----------------------------------------------------------------------
2022-05-18T04:14:58.4096708Z Ran 1 test in 1.191s
2022-05-18T04:14:58.4096822Z 
2022-05-18T04:14:58.4096886Z OK
2022-05-18T04:14:58.4096981Z 
2022-05-18T04:14:58.4097122Z Generating XML reports...
2022-05-18T04:14:58.4130872Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041457.xml
2022-05-18T04:14:59.1658731Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:14:59.1666926Z 
2022-05-18T04:14:59.1667154Z Running tests...
2022-05-18T04:14:59.1667734Z ----------------------------------------------------------------------
2022-05-18T04:14:59.4438569Z   test_allgather_basics_cuda (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5450
2022-05-18T04:14:59.4459087Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5451
2022-05-18T04:14:59.4481821Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5452
2022-05-18T04:14:59.4505389Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5453
2022-05-18T04:15:00.0826744Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:15:00.0946368Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:15:00.1231181Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:15:00.1376705Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:15:00.3565678Z skip: Need at least 2 CUDA devices (1.187s)
2022-05-18T04:15:00.3566013Z 
2022-05-18T04:15:00.3566372Z ----------------------------------------------------------------------
2022-05-18T04:15:00.3566622Z Ran 1 test in 1.187s
2022-05-18T04:15:00.3566722Z 
2022-05-18T04:15:00.3566825Z OK (skipped=1)
2022-05-18T04:15:00.3566953Z 
2022-05-18T04:15:00.3567036Z Generating XML reports...
2022-05-18T04:15:00.3579687Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041459.xml
2022-05-18T04:15:01.1154164Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:15:01.1164043Z 
2022-05-18T04:15:01.1164383Z Running tests...
2022-05-18T04:15:01.1164814Z ----------------------------------------------------------------------
2022-05-18T04:15:01.4070338Z   test_allgather_checks (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5505
2022-05-18T04:15:01.4091278Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5506
2022-05-18T04:15:01.4113638Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5507
2022-05-18T04:15:01.4136961Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5508
2022-05-18T04:15:02.0439393Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:15:02.0580067Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:15:02.0611908Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:15:02.0736348Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:15:02.3168422Z ok (1.200s)
2022-05-18T04:15:02.3168704Z 
2022-05-18T04:15:02.3169140Z ----------------------------------------------------------------------
2022-05-18T04:15:02.3169407Z Ran 1 test in 1.200s
2022-05-18T04:15:02.3169524Z 
2022-05-18T04:15:02.3169587Z OK
2022-05-18T04:15:02.3169679Z 
2022-05-18T04:15:02.3169773Z Generating XML reports...
2022-05-18T04:15:02.3203327Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041501.xml
2022-05-18T04:15:03.0854073Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:15:03.0862743Z 
2022-05-18T04:15:03.0863003Z Running tests...
2022-05-18T04:15:03.0863429Z ----------------------------------------------------------------------
2022-05-18T04:15:03.3672355Z   test_allgather_coalesced_async (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5572
2022-05-18T04:15:03.3693605Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5573
2022-05-18T04:15:03.3716077Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5574
2022-05-18T04:15:03.3739485Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5575
2022-05-18T04:15:03.9682504Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:15:04.0093551Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:15:04.0296963Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:15:04.0313128Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:15:04.0623521Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:15:04.0624118Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:15:04.0624768Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:15:04.0625287Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T04:15:04.0626222Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:15:04.0627039Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:15:04.0627893Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:15:04.0725189Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:15:04.2768359Z ok (1.190s)
2022-05-18T04:15:04.2768616Z 
2022-05-18T04:15:04.2769073Z ----------------------------------------------------------------------
2022-05-18T04:15:04.2769474Z Ran 1 test in 1.190s
2022-05-18T04:15:04.2769656Z 
2022-05-18T04:15:04.2769751Z OK
2022-05-18T04:15:04.2769890Z 
2022-05-18T04:15:04.2770028Z Generating XML reports...
2022-05-18T04:15:04.2804584Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041503.xml
2022-05-18T04:15:05.0583338Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:15:05.0592869Z 
2022-05-18T04:15:05.0593285Z Running tests...
2022-05-18T04:15:05.0593694Z ----------------------------------------------------------------------
2022-05-18T04:15:05.3405133Z   test_allgather_coalesced_checks (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5639
2022-05-18T04:15:05.3426591Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5640
2022-05-18T04:15:05.3448699Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5641
2022-05-18T04:15:05.3471676Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5642
2022-05-18T04:15:06.0210273Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:15:06.0334280Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:15:06.0395274Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:15:06.0451044Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:15:06.2502040Z ok (1.191s)
2022-05-18T04:15:06.2502288Z 
2022-05-18T04:15:06.2502805Z ----------------------------------------------------------------------
2022-05-18T04:15:06.2503325Z Ran 1 test in 1.191s
2022-05-18T04:15:06.2503442Z 
2022-05-18T04:15:06.2503720Z OK
2022-05-18T04:15:06.2503814Z 
2022-05-18T04:15:06.2503909Z Generating XML reports...
2022-05-18T04:15:06.2536800Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041505.xml
2022-05-18T04:15:07.0041127Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:15:07.0049734Z 
2022-05-18T04:15:07.0049829Z Running tests...
2022-05-18T04:15:07.0050269Z ----------------------------------------------------------------------
2022-05-18T04:15:07.2853516Z   test_allgather_noncontiguous_input (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5706
2022-05-18T04:15:07.2876203Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5707
2022-05-18T04:15:07.2898524Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5708
2022-05-18T04:15:07.2921695Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5709
2022-05-18T04:15:07.9130857Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:15:07.9476164Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:15:07.9485520Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:15:07.9711150Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:15:08.1953026Z ok (1.190s)
2022-05-18T04:15:08.1953280Z 
2022-05-18T04:15:08.1953791Z ----------------------------------------------------------------------
2022-05-18T04:15:08.1954216Z Ran 1 test in 1.190s
2022-05-18T04:15:08.1954333Z 
2022-05-18T04:15:08.1954394Z OK
2022-05-18T04:15:08.1954473Z 
2022-05-18T04:15:08.1954576Z Generating XML reports...
2022-05-18T04:15:08.1987243Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041507.xml
2022-05-18T04:15:08.9656917Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:15:08.9666114Z 
2022-05-18T04:15:08.9666488Z Running tests...
2022-05-18T04:15:08.9666915Z ----------------------------------------------------------------------
2022-05-18T04:15:09.2500404Z   test_allgather_stress (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5773
2022-05-18T04:15:09.2521462Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5774
2022-05-18T04:15:09.2543337Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5775
2022-05-18T04:15:09.2566199Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5776
2022-05-18T04:15:09.8913961Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:15:09.9208588Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:15:09.9310597Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:15:09.9351450Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:15:11.6623242Z ok (2.695s)
2022-05-18T04:15:11.6623504Z 
2022-05-18T04:15:11.6623956Z ----------------------------------------------------------------------
2022-05-18T04:15:11.6624352Z Ran 1 test in 2.696s
2022-05-18T04:15:11.6624530Z 
2022-05-18T04:15:11.6624621Z OK
2022-05-18T04:15:11.6624753Z 
2022-05-18T04:15:11.6624898Z Generating XML reports...
2022-05-18T04:15:11.6658804Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041508.xml
2022-05-18T04:15:12.4393089Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:15:12.4401745Z 
2022-05-18T04:15:12.4401855Z Running tests...
2022-05-18T04:15:12.4402340Z ----------------------------------------------------------------------
2022-05-18T04:15:12.7197091Z   test_allgather_stress_cuda (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5864
2022-05-18T04:15:12.7218231Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5865
2022-05-18T04:15:12.7239989Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5866
2022-05-18T04:15:12.7263461Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5867
2022-05-18T04:15:13.3157223Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:15:13.3678403Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:15:13.3924448Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:15:13.4390850Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:15:13.6293370Z skip: Need at least 2 CUDA devices (1.189s)
2022-05-18T04:15:13.6293641Z 
2022-05-18T04:15:13.6294159Z ----------------------------------------------------------------------
2022-05-18T04:15:13.6294511Z Ran 1 test in 1.189s
2022-05-18T04:15:13.6294625Z 
2022-05-18T04:15:13.6294688Z OK (skipped=1)
2022-05-18T04:15:13.6294799Z 
2022-05-18T04:15:13.6294887Z Generating XML reports...
2022-05-18T04:15:13.6328179Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041512.xml
2022-05-18T04:15:14.3837592Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:15:14.3846330Z 
2022-05-18T04:15:14.3846442Z Running tests...
2022-05-18T04:15:14.3846872Z ----------------------------------------------------------------------
2022-05-18T04:15:14.6646534Z   test_allreduce_basics (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5919
2022-05-18T04:15:14.6667878Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5920
2022-05-18T04:15:14.6689657Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5921
2022-05-18T04:15:14.6713653Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5922
2022-05-18T04:15:15.2944825Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:15:15.3226660Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:15:15.3495405Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:15:15.3511462Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:15:15.5744296Z ok (1.189s)
2022-05-18T04:15:15.5744597Z 
2022-05-18T04:15:15.5744968Z ----------------------------------------------------------------------
2022-05-18T04:15:15.5745216Z Ran 1 test in 1.190s
2022-05-18T04:15:15.5745353Z 
2022-05-18T04:15:15.5745421Z OK
2022-05-18T04:15:15.5745499Z 
2022-05-18T04:15:15.5745594Z Generating XML reports...
2022-05-18T04:15:15.5777556Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041514.xml
2022-05-18T04:15:16.3297677Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:15:16.3305962Z 
2022-05-18T04:15:16.3306132Z Running tests...
2022-05-18T04:15:16.3306474Z ----------------------------------------------------------------------
2022-05-18T04:15:16.6103456Z   test_allreduce_basics_cuda (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5986
2022-05-18T04:15:16.6125264Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5987
2022-05-18T04:15:16.6147222Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5988
2022-05-18T04:15:16.6170773Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5989
2022-05-18T04:15:17.2810178Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:15:17.2899661Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:15:17.2939689Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:15:17.3075896Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:15:17.5200294Z skip: Need at least 2 CUDA devices (1.189s)
2022-05-18T04:15:17.5200756Z 
2022-05-18T04:15:17.5201335Z ----------------------------------------------------------------------
2022-05-18T04:15:17.5201609Z Ran 1 test in 1.189s
2022-05-18T04:15:17.5201727Z 
2022-05-18T04:15:17.5201811Z OK (skipped=1)
2022-05-18T04:15:17.5201921Z 
2022-05-18T04:15:17.5202006Z Generating XML reports...
2022-05-18T04:15:17.5235646Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041516.xml
2022-05-18T04:15:18.2864682Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:15:18.2873851Z 
2022-05-18T04:15:18.2873968Z Running tests...
2022-05-18T04:15:18.2874558Z ----------------------------------------------------------------------
2022-05-18T04:15:18.5644565Z   test_allreduce_basics_cuda_using_work_api (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6041
2022-05-18T04:15:18.5665174Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6042
2022-05-18T04:15:18.5687406Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6043
2022-05-18T04:15:18.5710228Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6044
2022-05-18T04:15:19.1760596Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:15:19.2167671Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:15:19.2330492Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:15:19.2360790Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:15:19.3738866Z skip: Need at least 2 CUDA devices (1.086s)
2022-05-18T04:15:19.3739143Z 
2022-05-18T04:15:19.3739646Z ----------------------------------------------------------------------
2022-05-18T04:15:19.3740016Z Ran 1 test in 1.086s
2022-05-18T04:15:19.3740131Z 
2022-05-18T04:15:19.3740204Z OK (skipped=1)
2022-05-18T04:15:19.3740311Z 
2022-05-18T04:15:19.3773006Z Generating XML reports...
2022-05-18T04:15:19.3773557Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041518.xml
2022-05-18T04:15:20.1270396Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:15:20.1279029Z 
2022-05-18T04:15:20.1279139Z Running tests...
2022-05-18T04:15:20.1279546Z ----------------------------------------------------------------------
2022-05-18T04:15:20.4070990Z   test_allreduce_basics_using_work_api (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6096
2022-05-18T04:15:20.4092518Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6097
2022-05-18T04:15:20.4115379Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6098
2022-05-18T04:15:20.4138013Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6099
2022-05-18T04:15:21.0228613Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:15:21.0488947Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:15:21.0584586Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:15:21.0610711Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:15:21.3167473Z ok (1.189s)
2022-05-18T04:15:21.3167726Z 
2022-05-18T04:15:21.3168553Z ----------------------------------------------------------------------
2022-05-18T04:15:21.3168814Z Ran 1 test in 1.189s
2022-05-18T04:15:21.3168931Z 
2022-05-18T04:15:21.3168993Z OK
2022-05-18T04:15:21.3169084Z 
2022-05-18T04:15:21.3169166Z Generating XML reports...
2022-05-18T04:15:21.3202649Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041520.xml
2022-05-18T04:15:22.0768734Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:15:22.0777286Z 
2022-05-18T04:15:22.0777389Z Running tests...
2022-05-18T04:15:22.0777854Z ----------------------------------------------------------------------
2022-05-18T04:15:22.3673181Z   test_allreduce_checks (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6163
2022-05-18T04:15:22.3694830Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6164
2022-05-18T04:15:22.3717231Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6165
2022-05-18T04:15:22.3740904Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6166
2022-05-18T04:15:23.0071912Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:15:23.0176224Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:15:23.0182391Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:15:23.0328161Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:15:23.2770072Z ok (1.199s)
2022-05-18T04:15:23.2770321Z 
2022-05-18T04:15:23.2770845Z ----------------------------------------------------------------------
2022-05-18T04:15:23.2771273Z Ran 1 test in 1.199s
2022-05-18T04:15:23.2771391Z 
2022-05-18T04:15:23.2771456Z OK
2022-05-18T04:15:23.2771534Z 
2022-05-18T04:15:23.2771636Z Generating XML reports...
2022-05-18T04:15:23.2805396Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041522.xml
2022-05-18T04:15:24.0524792Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:15:24.0532886Z 
2022-05-18T04:15:24.0533043Z Running tests...
2022-05-18T04:15:24.0533498Z ----------------------------------------------------------------------
2022-05-18T04:15:24.3323796Z   test_allreduce_coalesced_async (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6230
2022-05-18T04:15:24.3345285Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6231
2022-05-18T04:15:24.3367194Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6232
2022-05-18T04:15:24.3390377Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6233
2022-05-18T04:15:24.9952573Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:15:25.0615151Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:15:25.0690204Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:15:25.0774606Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:15:25.0999021Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:15:25.1100553Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:15:25.1101413Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:15:25.1102811Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:15:25.1103972Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T04:15:25.1105354Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:15:25.1106208Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:15:25.1107078Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:15:25.3489647Z ok (1.295s)
2022-05-18T04:15:25.3489928Z 
2022-05-18T04:15:25.3490426Z ----------------------------------------------------------------------
2022-05-18T04:15:25.3490678Z Ran 1 test in 1.296s
2022-05-18T04:15:25.3490795Z 
2022-05-18T04:15:25.3490844Z OK
2022-05-18T04:15:25.3490952Z 
2022-05-18T04:15:25.3491051Z Generating XML reports...
2022-05-18T04:15:25.3524467Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041524.xml
2022-05-18T04:15:26.1079442Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:15:26.1087877Z 
2022-05-18T04:15:26.1088021Z Running tests...
2022-05-18T04:15:26.1088651Z ----------------------------------------------------------------------
2022-05-18T04:15:26.3866625Z   test_allreduce_coalesced_basics (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6297
2022-05-18T04:15:26.3887953Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6298
2022-05-18T04:15:26.3910654Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6299
2022-05-18T04:15:26.3933337Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6300
2022-05-18T04:15:27.0341858Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:15:27.0574210Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:15:27.1363101Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:15:27.1363664Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:15:27.3965579Z ok (1.287s)
2022-05-18T04:15:27.3965757Z 
2022-05-18T04:15:27.3966134Z ----------------------------------------------------------------------
2022-05-18T04:15:27.3966379Z Ran 1 test in 1.288s
2022-05-18T04:15:27.3966498Z 
2022-05-18T04:15:27.3966561Z OK
2022-05-18T04:15:27.3966658Z 
2022-05-18T04:15:27.3966756Z Generating XML reports...
2022-05-18T04:15:27.4000162Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041526.xml
2022-05-18T04:15:28.1516941Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:15:28.1525734Z 
2022-05-18T04:15:28.1525874Z Running tests...
2022-05-18T04:15:28.1526510Z ----------------------------------------------------------------------
2022-05-18T04:15:28.4385423Z   test_allreduce_coalesced_checks (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6364
2022-05-18T04:15:28.4406373Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6365
2022-05-18T04:15:28.4429178Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6366
2022-05-18T04:15:28.4453729Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6367
2022-05-18T04:15:29.1066352Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:15:29.1187360Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:15:29.1642165Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:15:29.1828032Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:15:29.4484347Z ok (1.296s)
2022-05-18T04:15:29.4484595Z 
2022-05-18T04:15:29.4485119Z ----------------------------------------------------------------------
2022-05-18T04:15:29.4485507Z Ran 1 test in 1.296s
2022-05-18T04:15:29.4485625Z 
2022-05-18T04:15:29.4485674Z OK
2022-05-18T04:15:29.4485766Z 
2022-05-18T04:15:29.4485861Z Generating XML reports...
2022-05-18T04:15:29.4519477Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041528.xml
2022-05-18T04:15:30.2084683Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:15:30.2093237Z 
2022-05-18T04:15:30.2093369Z Running tests...
2022-05-18T04:15:30.2094065Z ----------------------------------------------------------------------
2022-05-18T04:15:30.4909282Z   test_allreduce_coalesced_checks_cuda (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6431
2022-05-18T04:15:30.4930353Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6432
2022-05-18T04:15:30.4952492Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6433
2022-05-18T04:15:30.4975620Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6434
2022-05-18T04:15:31.1672075Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:15:31.1834394Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:15:31.1852538Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:15:31.2073646Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:15:31.4007322Z skip: Need at least 1 CUDA device (1.191s)
2022-05-18T04:15:31.4007675Z 
2022-05-18T04:15:31.4008088Z ----------------------------------------------------------------------
2022-05-18T04:15:31.4008353Z Ran 1 test in 1.191s
2022-05-18T04:15:31.4008455Z 
2022-05-18T04:15:31.4008532Z OK (skipped=1)
2022-05-18T04:15:31.4008640Z 
2022-05-18T04:15:31.4008726Z Generating XML reports...
2022-05-18T04:15:31.4042575Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041530.xml
2022-05-18T04:15:32.1592653Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:15:32.1601269Z 
2022-05-18T04:15:32.1601348Z Running tests...
2022-05-18T04:15:32.1602046Z ----------------------------------------------------------------------
2022-05-18T04:15:32.4457279Z   test_allreduce_coalesced_stress (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6486
2022-05-18T04:15:32.4477619Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6487
2022-05-18T04:15:32.4499401Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6488
2022-05-18T04:15:32.4522968Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6489
2022-05-18T04:15:33.0676770Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:15:33.0784250Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:15:33.0953535Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:15:33.0975839Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:15:34.2570259Z ok (2.097s)
2022-05-18T04:15:34.2570529Z 
2022-05-18T04:15:34.2571059Z ----------------------------------------------------------------------
2022-05-18T04:15:34.2571457Z Ran 1 test in 2.097s
2022-05-18T04:15:34.2571833Z 
2022-05-18T04:15:34.2571888Z OK
2022-05-18T04:15:34.2571981Z 
2022-05-18T04:15:34.2572077Z Generating XML reports...
2022-05-18T04:15:34.2605618Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041532.xml
2022-05-18T04:15:35.0412851Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:15:35.0421729Z 
2022-05-18T04:15:35.0421834Z Running tests...
2022-05-18T04:15:35.0422332Z ----------------------------------------------------------------------
2022-05-18T04:15:35.3233140Z   test_allreduce_stress (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6577
2022-05-18T04:15:35.3254321Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6578
2022-05-18T04:15:35.3277249Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6579
2022-05-18T04:15:35.3300284Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6580
2022-05-18T04:15:36.0484817Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:15:36.0562258Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:15:36.0624079Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:15:36.0876500Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:15:36.9341716Z ok (1.892s)
2022-05-18T04:15:36.9341983Z 
2022-05-18T04:15:36.9342449Z ----------------------------------------------------------------------
2022-05-18T04:15:36.9342704Z Ran 1 test in 1.892s
2022-05-18T04:15:36.9342821Z 
2022-05-18T04:15:36.9343004Z OK
2022-05-18T04:15:36.9343103Z 
2022-05-18T04:15:36.9343183Z Generating XML reports...
2022-05-18T04:15:36.9376678Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041535.xml
2022-05-18T04:15:37.7096699Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:15:37.7105501Z 
2022-05-18T04:15:37.7105812Z Running tests...
2022-05-18T04:15:37.7106447Z ----------------------------------------------------------------------
2022-05-18T04:15:37.9904800Z   test_allreduce_stress_cuda (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6668
2022-05-18T04:15:37.9926235Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6669
2022-05-18T04:15:37.9948425Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6670
2022-05-18T04:15:37.9972986Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6671
2022-05-18T04:15:38.6697343Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:15:38.6883206Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:15:38.7162602Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:15:38.7238211Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:15:38.9003207Z skip: Need at least 2 CUDA devices (1.189s)
2022-05-18T04:15:38.9003514Z 
2022-05-18T04:15:38.9003956Z ----------------------------------------------------------------------
2022-05-18T04:15:38.9004216Z Ran 1 test in 1.190s
2022-05-18T04:15:38.9004330Z 
2022-05-18T04:15:38.9004404Z OK (skipped=1)
2022-05-18T04:15:38.9004513Z 
2022-05-18T04:15:38.9004600Z Generating XML reports...
2022-05-18T04:15:38.9037478Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041537.xml
2022-05-18T04:15:39.6528129Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:15:39.6536874Z 
2022-05-18T04:15:39.6536988Z Running tests...
2022-05-18T04:15:39.6537920Z ----------------------------------------------------------------------
2022-05-18T04:15:39.9332820Z   test_barrier_implies_wait (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6723
2022-05-18T04:15:39.9353862Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6724
2022-05-18T04:15:39.9375965Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6725
2022-05-18T04:15:39.9399097Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6726
2022-05-18T04:15:40.6798614Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:15:40.6900021Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:15:40.7041045Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:15:40.7338865Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:15:41.0432120Z ok (1.389s)
2022-05-18T04:15:41.0432395Z 
2022-05-18T04:15:41.0432896Z ----------------------------------------------------------------------
2022-05-18T04:15:41.0433368Z Ran 1 test in 1.389s
2022-05-18T04:15:41.0433509Z 
2022-05-18T04:15:41.0433571Z OK
2022-05-18T04:15:41.0433664Z 
2022-05-18T04:15:41.0433758Z Generating XML reports...
2022-05-18T04:15:41.0467505Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041539.xml
2022-05-18T04:15:41.7992386Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:15:41.8000572Z 
2022-05-18T04:15:41.8000777Z Running tests...
2022-05-18T04:15:41.8001176Z ----------------------------------------------------------------------
2022-05-18T04:15:42.0794797Z   test_broadcast_basics (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6790
2022-05-18T04:15:42.0816493Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6791
2022-05-18T04:15:42.0838782Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6792
2022-05-18T04:15:42.0861876Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6793
2022-05-18T04:15:42.7573518Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:15:42.7868675Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:15:42.8419061Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:15:42.8652840Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:15:43.1893860Z ok (1.389s)
2022-05-18T04:15:43.1894104Z 
2022-05-18T04:15:43.1894598Z ----------------------------------------------------------------------
2022-05-18T04:15:43.1895075Z Ran 1 test in 1.389s
2022-05-18T04:15:43.1895229Z 
2022-05-18T04:15:43.1895292Z OK
2022-05-18T04:15:43.1895383Z 
2022-05-18T04:15:43.1895475Z Generating XML reports...
2022-05-18T04:15:43.1929250Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041541.xml
2022-05-18T04:15:43.9550714Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:15:43.9559208Z 
2022-05-18T04:15:43.9559310Z Running tests...
2022-05-18T04:15:43.9560397Z ----------------------------------------------------------------------
2022-05-18T04:15:44.2343608Z   test_broadcast_basics_cuda (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6857
2022-05-18T04:15:44.2363380Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6858
2022-05-18T04:15:44.2385522Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6859
2022-05-18T04:15:44.2408970Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6860
2022-05-18T04:15:44.8935155Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:15:44.9218821Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:15:44.9421189Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:15:44.9854037Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:15:45.1438409Z skip: Need at least 2 CUDA devices (1.188s)
2022-05-18T04:15:45.1438698Z 
2022-05-18T04:15:45.1439321Z ----------------------------------------------------------------------
2022-05-18T04:15:45.1439583Z Ran 1 test in 1.188s
2022-05-18T04:15:45.1439699Z 
2022-05-18T04:15:45.1439774Z OK (skipped=1)
2022-05-18T04:15:45.1439883Z 
2022-05-18T04:15:45.1439973Z Generating XML reports...
2022-05-18T04:15:45.1473586Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041543.xml
2022-05-18T04:15:45.9044589Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:15:45.9053423Z 
2022-05-18T04:15:45.9053547Z Running tests...
2022-05-18T04:15:45.9054351Z ----------------------------------------------------------------------
2022-05-18T04:15:46.1867137Z   test_broadcast_checks (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6912
2022-05-18T04:15:46.1889498Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6913
2022-05-18T04:15:46.1911785Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6914
2022-05-18T04:15:46.1934755Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6915
2022-05-18T04:15:46.8593713Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:15:46.8824158Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:15:46.8874206Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:15:46.8924911Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:15:47.2050073Z ok (1.299s)
2022-05-18T04:15:47.2050368Z 
2022-05-18T04:15:47.2050805Z ----------------------------------------------------------------------
2022-05-18T04:15:47.2051073Z Ran 1 test in 1.300s
2022-05-18T04:15:47.2051190Z 
2022-05-18T04:15:47.2051253Z OK
2022-05-18T04:15:47.2051334Z 
2022-05-18T04:15:47.2051426Z Generating XML reports...
2022-05-18T04:15:47.2085653Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041545.xml
2022-05-18T04:15:47.9738203Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:15:47.9746717Z 
2022-05-18T04:15:47.9746864Z Running tests...
2022-05-18T04:15:47.9747448Z ----------------------------------------------------------------------
2022-05-18T04:15:48.2535155Z   test_broadcast_stress (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6979
2022-05-18T04:15:48.2556686Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6980
2022-05-18T04:15:48.2579027Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6981
2022-05-18T04:15:48.2603370Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6982
2022-05-18T04:15:48.8680604Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:15:48.8763687Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:15:48.8948747Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:15:48.9021529Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:15:49.5640531Z ok (1.589s)
2022-05-18T04:15:49.5641461Z 
2022-05-18T04:15:49.5642046Z ----------------------------------------------------------------------
2022-05-18T04:15:49.5642315Z Ran 1 test in 1.589s
2022-05-18T04:15:49.5642434Z 
2022-05-18T04:15:49.5642483Z OK
2022-05-18T04:15:49.5642577Z 
2022-05-18T04:15:49.5642675Z Generating XML reports...
2022-05-18T04:15:49.5675303Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041547.xml
2022-05-18T04:15:50.3373507Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:15:50.3381316Z 
2022-05-18T04:15:50.3381422Z Running tests...
2022-05-18T04:15:50.3382026Z ----------------------------------------------------------------------
2022-05-18T04:15:50.6178444Z   test_broadcast_stress_cuda (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7070
2022-05-18T04:15:50.6199955Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7071
2022-05-18T04:15:50.6222675Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7072
2022-05-18T04:15:50.6246291Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7073
2022-05-18T04:15:51.3313285Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:15:51.3426314Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:15:51.3567137Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:15:51.3928393Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:15:51.5276660Z skip: Need at least 2 CUDA devices (1.189s)
2022-05-18T04:15:51.5276977Z 
2022-05-18T04:15:51.5277491Z ----------------------------------------------------------------------
2022-05-18T04:15:51.5277879Z Ran 1 test in 1.189s
2022-05-18T04:15:51.5277994Z 
2022-05-18T04:15:51.5278084Z OK (skipped=1)
2022-05-18T04:15:51.5278189Z 
2022-05-18T04:15:51.5278283Z Generating XML reports...
2022-05-18T04:15:51.5311547Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041550.xml
2022-05-18T04:15:52.2763654Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:15:52.2772815Z 
2022-05-18T04:15:52.2772954Z Running tests...
2022-05-18T04:15:52.2773443Z ----------------------------------------------------------------------
2022-05-18T04:15:52.5575472Z   test_empty_tensors (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7125
2022-05-18T04:15:52.5596746Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7126
2022-05-18T04:15:52.5618588Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7127
2022-05-18T04:15:52.5643001Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7128
2022-05-18T04:15:53.1442091Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:15:53.1442677Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:15:53.1443098Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:15:53.1481187Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:15:53.3670825Z ok (1.089s)
2022-05-18T04:15:53.3671145Z 
2022-05-18T04:15:53.3671579Z ----------------------------------------------------------------------
2022-05-18T04:15:53.3671845Z Ran 1 test in 1.090s
2022-05-18T04:15:53.3671965Z 
2022-05-18T04:15:53.3672029Z OK
2022-05-18T04:15:53.3672107Z 
2022-05-18T04:15:53.3672203Z Generating XML reports...
2022-05-18T04:15:53.3705205Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041552.xml
2022-05-18T04:15:54.1203999Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:15:54.1212149Z 
2022-05-18T04:15:54.1212276Z Running tests...
2022-05-18T04:15:54.1212841Z ----------------------------------------------------------------------
2022-05-18T04:15:54.3998926Z   test_gather_basics (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7192
2022-05-18T04:15:54.4021223Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7193
2022-05-18T04:15:54.4044185Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7194
2022-05-18T04:15:54.4068676Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7195
2022-05-18T04:15:55.0476157Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:15:55.0498037Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:15:55.0596949Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:15:55.0600175Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:15:55.3098014Z ok (1.188s)
2022-05-18T04:15:55.3098188Z 
2022-05-18T04:15:55.3098529Z ----------------------------------------------------------------------
2022-05-18T04:15:55.3098847Z Ran 1 test in 1.188s
2022-05-18T04:15:55.3098950Z 
2022-05-18T04:15:55.3099010Z OK
2022-05-18T04:15:55.3099100Z 
2022-05-18T04:15:55.3099193Z Generating XML reports...
2022-05-18T04:15:55.3133470Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041554.xml
2022-05-18T04:15:56.0767380Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:15:56.0775769Z 
2022-05-18T04:15:56.0775903Z Running tests...
2022-05-18T04:15:56.0776469Z ----------------------------------------------------------------------
2022-05-18T04:15:56.3551946Z   test_gather_basics_cuda (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7259
2022-05-18T04:15:56.3574387Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7260
2022-05-18T04:15:56.3597294Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7261
2022-05-18T04:15:56.3621667Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7262
2022-05-18T04:15:57.0489219Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:15:57.0690735Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:15:57.1187160Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:15:57.1248216Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:15:57.2651256Z skip: Need at least 2 CUDA devices (1.187s)
2022-05-18T04:15:57.2651570Z 
2022-05-18T04:15:57.2652132Z ----------------------------------------------------------------------
2022-05-18T04:15:57.2652376Z Ran 1 test in 1.187s
2022-05-18T04:15:57.2652494Z 
2022-05-18T04:15:57.2652569Z OK (skipped=1)
2022-05-18T04:15:57.2652678Z 
2022-05-18T04:15:57.2652765Z Generating XML reports...
2022-05-18T04:15:57.2688067Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041556.xml
2022-05-18T04:15:58.0363805Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:15:58.0373433Z 
2022-05-18T04:15:58.0373741Z Running tests...
2022-05-18T04:15:58.0374371Z ----------------------------------------------------------------------
2022-05-18T04:15:58.3204672Z   test_gather_checks (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7314
2022-05-18T04:15:58.3225690Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7315
2022-05-18T04:15:58.3247941Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7316
2022-05-18T04:15:58.3271741Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7317
2022-05-18T04:15:58.9832417Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:15:58.9881392Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:15:58.9884013Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:15:58.9894552Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:15:59.2300804Z ok (1.192s)
2022-05-18T04:15:59.2301010Z 
2022-05-18T04:15:59.2301528Z ----------------------------------------------------------------------
2022-05-18T04:15:59.2301958Z Ran 1 test in 1.193s
2022-05-18T04:15:59.2302073Z 
2022-05-18T04:15:59.2302133Z OK
2022-05-18T04:15:59.2302222Z 
2022-05-18T04:15:59.2302313Z Generating XML reports...
2022-05-18T04:15:59.2336204Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041558.xml
2022-05-18T04:15:59.9877046Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:15:59.9886321Z 
2022-05-18T04:15:59.9886400Z Running tests...
2022-05-18T04:15:59.9887353Z ----------------------------------------------------------------------
2022-05-18T04:16:00.2687550Z   test_gather_noncontiguous_input (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7381
2022-05-18T04:16:00.2709274Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7382
2022-05-18T04:16:00.2731321Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7383
2022-05-18T04:16:00.2754741Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7384
2022-05-18T04:16:00.9570041Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:16:00.9570647Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:16:00.9579479Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:16:00.9579968Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:16:01.1786664Z ok (1.190s)
2022-05-18T04:16:01.1786933Z 
2022-05-18T04:16:01.1787416Z ----------------------------------------------------------------------
2022-05-18T04:16:01.1787671Z Ran 1 test in 1.190s
2022-05-18T04:16:01.1787787Z 
2022-05-18T04:16:01.1787849Z OK
2022-05-18T04:16:01.1787941Z 
2022-05-18T04:16:01.1788038Z Generating XML reports...
2022-05-18T04:16:01.1820587Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041559.xml
2022-05-18T04:16:01.9471629Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:16:01.9479962Z 
2022-05-18T04:16:01.9480095Z Running tests...
2022-05-18T04:16:01.9480682Z ----------------------------------------------------------------------
2022-05-18T04:16:02.2277081Z   test_gather_stress (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7448
2022-05-18T04:16:02.2299376Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7449
2022-05-18T04:16:02.2322268Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7450
2022-05-18T04:16:02.2345713Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7451
2022-05-18T04:16:02.9202266Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:16:02.9377178Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:16:02.9854262Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:16:03.0117418Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:16:05.0480399Z ok (3.100s)
2022-05-18T04:16:05.0480623Z 
2022-05-18T04:16:05.0480971Z ----------------------------------------------------------------------
2022-05-18T04:16:05.0481319Z Ran 1 test in 3.100s
2022-05-18T04:16:05.0481433Z 
2022-05-18T04:16:05.0481480Z OK
2022-05-18T04:16:05.0481571Z 
2022-05-18T04:16:05.0481666Z Generating XML reports...
2022-05-18T04:16:05.0515348Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041601.xml
2022-05-18T04:16:05.8207198Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:16:05.8216320Z 
2022-05-18T04:16:05.8216450Z Running tests...
2022-05-18T04:16:05.8216858Z ----------------------------------------------------------------------
2022-05-18T04:16:06.1011306Z   test_gather_stress_cuda (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7539
2022-05-18T04:16:06.1032927Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7540
2022-05-18T04:16:06.1055841Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7541
2022-05-18T04:16:06.1078242Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7542
2022-05-18T04:16:06.7653651Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:16:06.7892892Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:16:06.8282134Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:16:06.8313557Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:16:07.0109125Z skip: Need at least 2 CUDA devices (1.189s)
2022-05-18T04:16:07.0109433Z 
2022-05-18T04:16:07.0118123Z ----------------------------------------------------------------------
2022-05-18T04:16:07.0118596Z Ran 1 test in 1.189s
2022-05-18T04:16:07.0118718Z 
2022-05-18T04:16:07.0118803Z OK (skipped=1)
2022-05-18T04:16:07.0118899Z 
2022-05-18T04:16:07.0118988Z Generating XML reports...
2022-05-18T04:16:07.0144368Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041605.xml
2022-05-18T04:16:07.7659211Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:16:07.7667649Z 
2022-05-18T04:16:07.7667746Z Running tests...
2022-05-18T04:16:07.7668113Z ----------------------------------------------------------------------
2022-05-18T04:16:08.0439025Z   test_multi_device_constructor (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7594
2022-05-18T04:16:08.0460686Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7595
2022-05-18T04:16:08.0482904Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7596
2022-05-18T04:16:08.0505795Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7597
2022-05-18T04:16:08.6742297Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:16:08.7019574Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:16:08.7038759Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:16:08.7307543Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:16:09.0538062Z ok (1.287s)
2022-05-18T04:16:09.0538359Z 
2022-05-18T04:16:09.0538844Z ----------------------------------------------------------------------
2022-05-18T04:16:09.0539360Z Ran 1 test in 1.287s
2022-05-18T04:16:09.0539477Z 
2022-05-18T04:16:09.0539541Z OK
2022-05-18T04:16:09.0539619Z 
2022-05-18T04:16:09.0539777Z Generating XML reports...
2022-05-18T04:16:09.0572898Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041607.xml
2022-05-18T04:16:09.8136461Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:16:09.8145987Z 
2022-05-18T04:16:09.8146114Z Running tests...
2022-05-18T04:16:09.8146571Z ----------------------------------------------------------------------
2022-05-18T04:16:10.0975131Z   test_reduce_basics (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7665
2022-05-18T04:16:10.0997501Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7666
2022-05-18T04:16:10.1019751Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7667
2022-05-18T04:16:10.1043353Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7668
2022-05-18T04:16:10.7821329Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:16:10.8007313Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:16:10.8103512Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:16:10.8146815Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:16:11.1075844Z ok (1.293s)
2022-05-18T04:16:11.1076051Z 
2022-05-18T04:16:11.1076403Z ----------------------------------------------------------------------
2022-05-18T04:16:11.1076655Z Ran 1 test in 1.293s
2022-05-18T04:16:11.1076772Z 
2022-05-18T04:16:11.1076836Z OK
2022-05-18T04:16:11.1076928Z 
2022-05-18T04:16:11.1077020Z Generating XML reports...
2022-05-18T04:16:11.1110667Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041609.xml
2022-05-18T04:16:11.8639817Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:16:11.8649683Z 
2022-05-18T04:16:11.8650336Z Running tests...
2022-05-18T04:16:11.8650989Z ----------------------------------------------------------------------
2022-05-18T04:16:12.1459522Z   test_reduce_basics_cuda (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7732
2022-05-18T04:16:12.1481377Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7733
2022-05-18T04:16:12.1503466Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7734
2022-05-18T04:16:12.1528034Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7735
2022-05-18T04:16:12.8228298Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:16:12.8300499Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:16:12.8408172Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:16:12.8923813Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:16:13.0558361Z skip: Need at least 2 CUDA devices (1.191s)
2022-05-18T04:16:13.0558618Z 
2022-05-18T04:16:13.0559084Z ----------------------------------------------------------------------
2022-05-18T04:16:13.0559477Z Ran 1 test in 1.191s
2022-05-18T04:16:13.0559659Z 
2022-05-18T04:16:13.0559772Z OK (skipped=1)
2022-05-18T04:16:13.0559937Z 
2022-05-18T04:16:13.0560070Z Generating XML reports...
2022-05-18T04:16:13.0594821Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041611.xml
2022-05-18T04:16:13.8170978Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:16:13.8180361Z 
2022-05-18T04:16:13.8180573Z Running tests...
2022-05-18T04:16:13.8180921Z ----------------------------------------------------------------------
2022-05-18T04:16:14.0977357Z   test_reduce_checks (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7787
2022-05-18T04:16:14.0997969Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7788
2022-05-18T04:16:14.1020044Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7789
2022-05-18T04:16:14.1044245Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7790
2022-05-18T04:16:14.7344215Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:16:14.7516004Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:16:14.7636605Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:16:14.7814803Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:16:15.0073461Z ok (1.189s)
2022-05-18T04:16:15.0073753Z 
2022-05-18T04:16:15.0074200Z ----------------------------------------------------------------------
2022-05-18T04:16:15.0074450Z Ran 1 test in 1.189s
2022-05-18T04:16:15.0074560Z 
2022-05-18T04:16:15.0074623Z OK
2022-05-18T04:16:15.0074717Z 
2022-05-18T04:16:15.0074811Z Generating XML reports...
2022-05-18T04:16:15.0110346Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041613.xml
2022-05-18T04:16:15.7712766Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:16:15.7722037Z 
2022-05-18T04:16:15.7722400Z Running tests...
2022-05-18T04:16:15.7722817Z ----------------------------------------------------------------------
2022-05-18T04:16:16.0519548Z   test_reduce_stress (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7854
2022-05-18T04:16:16.0540874Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7855
2022-05-18T04:16:16.0563372Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7856
2022-05-18T04:16:16.0587185Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7857
2022-05-18T04:16:16.7065893Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:16:16.7118777Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:16:16.7433474Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:16:16.7739442Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:16:18.4642436Z ok (2.692s)
2022-05-18T04:16:18.4642687Z 
2022-05-18T04:16:18.4643266Z ----------------------------------------------------------------------
2022-05-18T04:16:18.4643600Z Ran 1 test in 2.692s
2022-05-18T04:16:18.4643718Z 
2022-05-18T04:16:18.4643766Z OK
2022-05-18T04:16:18.4643859Z 
2022-05-18T04:16:18.4643967Z Generating XML reports...
2022-05-18T04:16:18.4677030Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041615.xml
2022-05-18T04:16:19.2406917Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:16:19.2415343Z 
2022-05-18T04:16:19.2415482Z Running tests...
2022-05-18T04:16:19.2416078Z ----------------------------------------------------------------------
2022-05-18T04:16:19.5200865Z   test_reduce_stress_cuda (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7945
2022-05-18T04:16:19.5221822Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7946
2022-05-18T04:16:19.5243912Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7947
2022-05-18T04:16:19.5268430Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7948
2022-05-18T04:16:20.1806717Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:16:20.1998455Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:16:20.3061190Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:16:20.3069006Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:16:20.4298061Z skip: Need at least 2 CUDA devices (1.188s)
2022-05-18T04:16:20.4298340Z 
2022-05-18T04:16:20.4298769Z ----------------------------------------------------------------------
2022-05-18T04:16:20.4299148Z Ran 1 test in 1.188s
2022-05-18T04:16:20.4299333Z 
2022-05-18T04:16:20.4299411Z OK (skipped=1)
2022-05-18T04:16:20.4299552Z 
2022-05-18T04:16:20.4299711Z Generating XML reports...
2022-05-18T04:16:20.4334554Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041619.xml
2022-05-18T04:16:21.2107992Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:16:21.2117711Z 
2022-05-18T04:16:21.2118016Z Running tests...
2022-05-18T04:16:21.2118669Z ----------------------------------------------------------------------
2022-05-18T04:16:21.4912955Z   test_round_robin (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8000
2022-05-18T04:16:21.4934731Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8001
2022-05-18T04:16:21.4956693Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8002
2022-05-18T04:16:21.4980778Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8003
2022-05-18T04:16:22.1486391Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:16:22.1549991Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:16:22.1691839Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:16:22.1744249Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:16:22.5012465Z ok (1.289s)
2022-05-18T04:16:22.5012704Z 
2022-05-18T04:16:22.5013293Z ----------------------------------------------------------------------
2022-05-18T04:16:22.5013616Z Ran 1 test in 1.289s
2022-05-18T04:16:22.5013787Z 
2022-05-18T04:16:22.5013912Z OK
2022-05-18T04:16:22.5014088Z 
2022-05-18T04:16:22.5014266Z Generating XML reports...
2022-05-18T04:16:22.5048021Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041621.xml
2022-05-18T04:16:23.2631465Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:16:23.2639213Z 
2022-05-18T04:16:23.2639352Z Running tests...
2022-05-18T04:16:23.2639952Z ----------------------------------------------------------------------
2022-05-18T04:16:23.5469064Z   test_round_robin_create_destroy (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8079
2022-05-18T04:16:23.5490755Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8080
2022-05-18T04:16:23.5512743Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8081
2022-05-18T04:16:23.5536557Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8082
2022-05-18T04:16:24.2619030Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:16:24.2762299Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:16:24.2856084Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:16:24.3011718Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:16:24.7571923Z ok (1.493s)
2022-05-18T04:16:24.7572302Z 
2022-05-18T04:16:24.7572624Z ----------------------------------------------------------------------
2022-05-18T04:16:24.7572878Z Ran 1 test in 1.493s
2022-05-18T04:16:24.7573007Z 
2022-05-18T04:16:24.7573069Z OK
2022-05-18T04:16:24.7573163Z 
2022-05-18T04:16:24.7573244Z Generating XML reports...
2022-05-18T04:16:24.7607677Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041623.xml
2022-05-18T04:16:25.5258307Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:16:25.5267491Z 
2022-05-18T04:16:25.5267612Z Running tests...
2022-05-18T04:16:25.5268213Z ----------------------------------------------------------------------
2022-05-18T04:16:25.8104169Z   test_scatter_basics (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8182
2022-05-18T04:16:25.8125817Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8183
2022-05-18T04:16:25.8147725Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8184
2022-05-18T04:16:25.8171020Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8185
2022-05-18T04:16:26.4361992Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:16:26.4477021Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:16:26.4577793Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:16:26.4656290Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:16:26.7200458Z ok (1.193s)
2022-05-18T04:16:26.7200672Z 
2022-05-18T04:16:26.7201051Z ----------------------------------------------------------------------
2022-05-18T04:16:26.7201294Z Ran 1 test in 1.193s
2022-05-18T04:16:26.7201416Z 
2022-05-18T04:16:26.7201485Z OK
2022-05-18T04:16:26.7201587Z 
2022-05-18T04:16:26.7201681Z Generating XML reports...
2022-05-18T04:16:26.7238227Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041625.xml
2022-05-18T04:16:27.4985929Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:16:27.4995267Z 
2022-05-18T04:16:27.4995373Z Running tests...
2022-05-18T04:16:27.4995925Z ----------------------------------------------------------------------
2022-05-18T04:16:27.7828304Z   test_scatter_basics_cuda (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8249
2022-05-18T04:16:27.7851301Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8250
2022-05-18T04:16:27.7873663Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8251
2022-05-18T04:16:27.7897464Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8252
2022-05-18T04:16:28.5108292Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:16:28.5197741Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:16:28.5246495Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:16:28.5609051Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:16:28.6928380Z skip: Need at least 2 CUDA devices (1.193s)
2022-05-18T04:16:28.6928695Z 
2022-05-18T04:16:28.6929190Z ----------------------------------------------------------------------
2022-05-18T04:16:28.6929463Z Ran 1 test in 1.193s
2022-05-18T04:16:28.6929564Z 
2022-05-18T04:16:28.6929638Z OK (skipped=1)
2022-05-18T04:16:28.6929943Z 
2022-05-18T04:16:28.6930031Z Generating XML reports...
2022-05-18T04:16:28.6962057Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041627.xml
2022-05-18T04:16:29.4637912Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:16:29.4646515Z 
2022-05-18T04:16:29.4646741Z Running tests...
2022-05-18T04:16:29.4647106Z ----------------------------------------------------------------------
2022-05-18T04:16:29.7514799Z   test_scatter_checks (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8304
2022-05-18T04:16:29.7536983Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8305
2022-05-18T04:16:29.7559919Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8306
2022-05-18T04:16:29.7584135Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8307
2022-05-18T04:16:30.4217254Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:16:30.4469877Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:16:30.4512866Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:16:30.4770241Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:16:30.7616110Z ok (1.297s)
2022-05-18T04:16:30.7616323Z 
2022-05-18T04:16:30.7616776Z ----------------------------------------------------------------------
2022-05-18T04:16:30.7617190Z Ran 1 test in 1.297s
2022-05-18T04:16:30.7617371Z 
2022-05-18T04:16:30.7617450Z OK
2022-05-18T04:16:30.7617591Z 
2022-05-18T04:16:30.7617743Z Generating XML reports...
2022-05-18T04:16:30.7652310Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041629.xml
2022-05-18T04:16:31.5380461Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:16:31.5390251Z 
2022-05-18T04:16:31.5390501Z Running tests...
2022-05-18T04:16:31.5391028Z ----------------------------------------------------------------------
2022-05-18T04:16:31.8232766Z   test_scatter_stress (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8371
2022-05-18T04:16:31.8255339Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8372
2022-05-18T04:16:31.8277265Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8373
2022-05-18T04:16:31.8300409Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8374
2022-05-18T04:16:32.4405535Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:16:32.4563573Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:16:32.4757560Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:16:32.4831677Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:16:34.4360842Z ok (2.897s)
2022-05-18T04:16:34.4360999Z 
2022-05-18T04:16:34.4361324Z ----------------------------------------------------------------------
2022-05-18T04:16:34.4361620Z Ran 1 test in 2.897s
2022-05-18T04:16:34.4361739Z 
2022-05-18T04:16:34.4361801Z OK
2022-05-18T04:16:34.4361894Z 
2022-05-18T04:16:34.4361986Z Generating XML reports...
2022-05-18T04:16:34.4396308Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041631.xml
2022-05-18T04:16:35.2318125Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:16:35.2327076Z 
2022-05-18T04:16:35.2327635Z Running tests...
2022-05-18T04:16:35.2328038Z ----------------------------------------------------------------------
2022-05-18T04:16:35.2334185Z   test_scatter_stress_cuda (__main__.ProcessGroupGlooTest) ... skip: Test is flaky, see https://github.com/pytorch/pytorch/issues/15963 (0.001s)
2022-05-18T04:16:35.2334657Z 
2022-05-18T04:16:35.2335231Z ----------------------------------------------------------------------
2022-05-18T04:16:35.2335487Z Ran 1 test in 0.001s
2022-05-18T04:16:35.2335601Z 
2022-05-18T04:16:35.2335660Z OK (skipped=1)
2022-05-18T04:16:35.2335767Z 
2022-05-18T04:16:35.2335875Z Generating XML reports...
2022-05-18T04:16:35.2360261Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041635.xml
2022-05-18T04:16:35.9099717Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:16:35.9108792Z 
2022-05-18T04:16:35.9108895Z Running tests...
2022-05-18T04:16:35.9109287Z ----------------------------------------------------------------------
2022-05-18T04:16:36.1947299Z   test_send_recv_all_to_all (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8472
2022-05-18T04:16:36.1969001Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8473
2022-05-18T04:16:36.1991785Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8474
2022-05-18T04:16:36.2014859Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8475
2022-05-18T04:16:36.8601359Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:16:36.8801681Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:16:36.8807196Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:16:36.8887199Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:16:37.1044939Z ok (1.193s)
2022-05-18T04:16:37.1045147Z 
2022-05-18T04:16:37.1045484Z ----------------------------------------------------------------------
2022-05-18T04:16:37.1045769Z Ran 1 test in 1.194s
2022-05-18T04:16:37.1045884Z 
2022-05-18T04:16:37.1045947Z OK
2022-05-18T04:16:37.1046040Z 
2022-05-18T04:16:37.1046150Z Generating XML reports...
2022-05-18T04:16:37.1086346Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041635.xml
2022-05-18T04:16:37.8826921Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:16:37.8835857Z 
2022-05-18T04:16:37.8836037Z Running tests...
2022-05-18T04:16:37.8836388Z ----------------------------------------------------------------------
2022-05-18T04:16:37.8840271Z   test_sparse_allreduce_basics (__main__.ProcessGroupGlooTest) ... skip: intermittent failures on Windows, in CI (0.000s)
2022-05-18T04:16:37.8840630Z 
2022-05-18T04:16:37.8841030Z ----------------------------------------------------------------------
2022-05-18T04:16:37.8841454Z Ran 1 test in 0.000s
2022-05-18T04:16:37.8841585Z 
2022-05-18T04:16:37.8841663Z OK (skipped=1)
2022-05-18T04:16:37.8841757Z 
2022-05-18T04:16:37.8841849Z Generating XML reports...
2022-05-18T04:16:37.8866904Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041637.xml
2022-05-18T04:16:38.5572798Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:16:38.5581670Z 
2022-05-18T04:16:38.5581861Z Running tests...
2022-05-18T04:16:38.5582297Z ----------------------------------------------------------------------
2022-05-18T04:16:38.8378492Z   test_sparse_allreduce_basics_cuda (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8549
2022-05-18T04:16:38.8400457Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8550
2022-05-18T04:16:38.8423473Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8551
2022-05-18T04:16:38.8446432Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8552
2022-05-18T04:16:39.5578651Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:16:39.5697378Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:16:39.5863840Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:16:39.6299531Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:16:39.8478109Z skip: Need at least 2 CUDA devices (1.289s)
2022-05-18T04:16:39.8478400Z 
2022-05-18T04:16:39.8478776Z ----------------------------------------------------------------------
2022-05-18T04:16:39.8479033Z Ran 1 test in 1.290s
2022-05-18T04:16:39.8479148Z 
2022-05-18T04:16:39.8479223Z OK (skipped=1)
2022-05-18T04:16:39.8479335Z 
2022-05-18T04:16:39.8479421Z Generating XML reports...
2022-05-18T04:16:39.8513429Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041638.xml
2022-05-18T04:16:40.6141182Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:16:40.6150707Z 
2022-05-18T04:16:40.6151005Z Running tests...
2022-05-18T04:16:40.6151641Z ----------------------------------------------------------------------
2022-05-18T04:16:40.8982173Z   test_sparse_allreduce_checks (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8604
2022-05-18T04:16:40.9003691Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8605
2022-05-18T04:16:40.9025905Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8606
2022-05-18T04:16:40.9049282Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8607
2022-05-18T04:16:41.5667231Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:16:41.5878457Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:16:41.6053803Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:16:41.6100343Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:16:41.9129707Z ok (1.298s)
2022-05-18T04:16:41.9129989Z 
2022-05-18T04:16:41.9130548Z ----------------------------------------------------------------------
2022-05-18T04:16:41.9130935Z Ran 1 test in 1.298s
2022-05-18T04:16:41.9131051Z 
2022-05-18T04:16:41.9131111Z OK
2022-05-18T04:16:41.9131190Z 
2022-05-18T04:16:41.9131283Z Generating XML reports...
2022-05-18T04:16:41.9167263Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041640.xml
2022-05-18T04:16:42.6917497Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:16:42.6926725Z 
2022-05-18T04:16:42.6926888Z Running tests...
2022-05-18T04:16:42.6927524Z ----------------------------------------------------------------------
2022-05-18T04:16:42.6970790Z   test_forward_backward (__main__.ReducerTest) ... ok (0.004s)
2022-05-18T04:16:42.7041990Z 
2022-05-18T04:16:42.7042467Z ----------------------------------------------------------------------
2022-05-18T04:16:42.7042976Z Ran 1 test in 0.011s
2022-05-18T04:16:42.7043163Z 
2022-05-18T04:16:42.7043227Z OK
2022-05-18T04:16:42.7043325Z 
2022-05-18T04:16:42.7043418Z Generating XML reports...
2022-05-18T04:16:42.7067148Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ReducerTest-20220518041642.xml
2022-05-18T04:16:43.3777053Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:16:43.3785851Z 
2022-05-18T04:16:43.3785962Z Running tests...
2022-05-18T04:16:43.3786418Z ----------------------------------------------------------------------
2022-05-18T04:16:43.3841203Z   test_forward_backward_optimizer (__main__.ReducerTest) ... [W reducer.cpp:1258] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-05-18T04:16:43.3854835Z ok (0.007s)
2022-05-18T04:16:43.3903393Z 
2022-05-18T04:16:43.3904176Z ----------------------------------------------------------------------
2022-05-18T04:16:43.3904595Z Ran 1 test in 0.012s
2022-05-18T04:16:43.3904775Z 
2022-05-18T04:16:43.3904855Z OK
2022-05-18T04:16:43.3905012Z 
2022-05-18T04:16:43.3905164Z Generating XML reports...
2022-05-18T04:16:43.3929995Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ReducerTest-20220518041643.xml
2022-05-18T04:16:44.0733976Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:16:44.0743955Z 
2022-05-18T04:16:44.0744285Z Running tests...
2022-05-18T04:16:44.0744934Z ----------------------------------------------------------------------
2022-05-18T04:16:44.0790279Z   test_forward_backward_unused_parameters (__main__.ReducerTest) ... ok (0.005s)
2022-05-18T04:16:44.0860698Z 
2022-05-18T04:16:44.0861146Z ----------------------------------------------------------------------
2022-05-18T04:16:44.0861532Z Ran 1 test in 0.012s
2022-05-18T04:16:44.0861647Z 
2022-05-18T04:16:44.0861710Z OK
2022-05-18T04:16:44.0861800Z 
2022-05-18T04:16:44.0861876Z Generating XML reports...
2022-05-18T04:16:44.0885672Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ReducerTest-20220518041644.xml
2022-05-18T04:16:44.7589065Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:16:44.7599183Z 
2022-05-18T04:16:44.7599332Z Running tests...
2022-05-18T04:16:44.7599804Z ----------------------------------------------------------------------
2022-05-18T04:16:44.7628348Z   test_multi_dtype_multi_bucket (__main__.ReducerTest) ... ok (0.003s)
2022-05-18T04:16:44.7715529Z 
2022-05-18T04:16:44.7715940Z ----------------------------------------------------------------------
2022-05-18T04:16:44.7716260Z Ran 1 test in 0.012s
2022-05-18T04:16:44.7716437Z 
2022-05-18T04:16:44.7716502Z OK
2022-05-18T04:16:44.7716626Z 
2022-05-18T04:16:44.7716732Z Generating XML reports...
2022-05-18T04:16:44.7753189Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ReducerTest-20220518041644.xml
2022-05-18T04:16:45.4565797Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:16:45.4574729Z 
2022-05-18T04:16:45.4574825Z Running tests...
2022-05-18T04:16:45.4575821Z ----------------------------------------------------------------------
2022-05-18T04:16:45.4626014Z   test_multi_dtype_single_bucket (__main__.ReducerTest) ... ok (0.005s)
2022-05-18T04:16:45.4689752Z 
2022-05-18T04:16:45.4690158Z ----------------------------------------------------------------------
2022-05-18T04:16:45.4690493Z Ran 1 test in 0.011s
2022-05-18T04:16:45.4690607Z 
2022-05-18T04:16:45.4690671Z OK
2022-05-18T04:16:45.4690764Z 
2022-05-18T04:16:45.4690863Z Generating XML reports...
2022-05-18T04:16:45.4715041Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ReducerTest-20220518041645.xml
2022-05-18T04:16:46.1377102Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:16:46.1386140Z 
2022-05-18T04:16:46.1386263Z Running tests...
2022-05-18T04:16:46.1386664Z ----------------------------------------------------------------------
2022-05-18T04:16:46.1409604Z   test_single_dtype_single_bucket (__main__.ReducerTest) ... ok (0.002s)
2022-05-18T04:16:46.1499246Z 
2022-05-18T04:16:46.1499878Z ----------------------------------------------------------------------
2022-05-18T04:16:46.1500259Z Ran 1 test in 0.011s
2022-05-18T04:16:46.1500388Z 
2022-05-18T04:16:46.1500436Z OK
2022-05-18T04:16:46.1500685Z 
2022-05-18T04:16:46.1500780Z Generating XML reports...
2022-05-18T04:16:46.1524216Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ReducerTest-20220518041646.xml
2022-05-18T04:16:46.8195477Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:16:46.8204092Z 
2022-05-18T04:16:46.8204180Z Running tests...
2022-05-18T04:16:46.8204652Z ----------------------------------------------------------------------
2022-05-18T04:16:47.0967856Z   test_logging_init (__main__.RendezvousEnvTest) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:16:47.0968464Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-05-18T04:16:47.1066228Z ok (0.286s)
2022-05-18T04:16:47.1066421Z 
2022-05-18T04:16:47.1066825Z ----------------------------------------------------------------------
2022-05-18T04:16:47.1067186Z Ran 1 test in 0.286s
2022-05-18T04:16:47.1067340Z 
2022-05-18T04:16:47.1067406Z OK
2022-05-18T04:16:47.1067521Z 
2022-05-18T04:16:47.1067625Z Generating XML reports...
2022-05-18T04:16:47.1092121Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-RendezvousEnvTest-20220518041646.xml
2022-05-18T04:16:47.8421704Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-05-18T04:16:47.8430513Z 
2022-05-18T04:16:47.8430634Z Running tests...
2022-05-18T04:16:47.8431233Z ----------------------------------------------------------------------
2022-05-18T04:16:48.1138370Z   test_default_store_timeout_gloo (__main__.TimeoutTest) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/74714 for allplatform(s) . If you're seeing this on your local machine and would like to enable this test, please make sure IN_CI is not set and you are not using the flag --import-disabled-tests. (0.270s)
2022-05-18T04:16:48.1138872Z 
2022-05-18T04:16:48.1139069Z ----------------------------------------------------------------------
2022-05-18T04:16:48.1139330Z Ran 1 test in 0.271s
2022-05-18T04:16:48.1139442Z 
2022-05-18T04:16:48.1139514Z OK (skipped=1)
2022-05-18T04:16:48.1139620Z 
2022-05-18T04:16:48.1139704Z Generating XML reports...
2022-05-18T04:16:48.1161873Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-TimeoutTest-20220518041647.xml
2022-05-18T04:16:48.3904141Z Running distributed/test_c10d_nccl ... [2022-05-18 04:16:48.390009]
2022-05-18T04:16:48.3904720Z Executing ['/opt/conda/bin/python', 'distributed/test_c10d_nccl.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-05-18 04:16:48.390092]
2022-05-18T04:16:48.9581341Z <unittest.suite.TestSuite tests=[<__main__.CommTest testMethod=test_all_reduce_coalesced_nccl>, <__main__.CommTest testMethod=test_broadcast_coalesced_nccl>, <__main__.CommTest testMethod=test_nccl_barrier>, <__main__.CommTest testMethod=test_nccl_barrier_device_ids>, <__main__.CommTest testMethod=test_nccl_barrier_device_ids_function_argument>, <__main__.CommTest testMethod=test_nccl_barrier_timeout>, <__main__.CommTest testMethod=test_nccl_barrier_timeout_new_group>, <__main__.CommTest testMethod=test_nccl_barrier_timeout_new_group_non_member>, <__main__.CommTest testMethod=test_nccl_warn_not_in_group_debug_detail>, <__main__.CommTest testMethod=test_nccl_warn_not_in_group_debug_info>, <__main__.CommTest testMethod=test_nccl_warn_not_in_group_debug_off>, <__main__.CommTest testMethod=test_pass_nccl_options_high_priority_stream>, <__main__.CommTest testMethod=test_sequence_num_incremented_nccl_default>, <__main__.CommTest testMethod=test_sequence_num_incremented_nccl_subgroup>, <__main__.CommTest testMethod=test_sequence_num_set_default_pg_nccl>, <__main__.CommTest testMethod=test_sequence_num_set_nccl_new_group>]>
2022-05-18T04:16:48.9583337Z test_all_reduce_coalesced_nccl (__main__.CommTest)
2022-05-18T04:16:48.9583651Z test_broadcast_coalesced_nccl (__main__.CommTest)
2022-05-18T04:16:48.9583882Z test_nccl_barrier (__main__.CommTest)
2022-05-18T04:16:48.9584108Z test_nccl_barrier_device_ids (__main__.CommTest)
2022-05-18T04:16:48.9584357Z test_nccl_barrier_device_ids_function_argument (__main__.CommTest)
2022-05-18T04:16:48.9584614Z test_nccl_barrier_timeout (__main__.CommTest)
2022-05-18T04:16:48.9584858Z test_nccl_barrier_timeout_new_group (__main__.CommTest)
2022-05-18T04:16:48.9585110Z test_nccl_barrier_timeout_new_group_non_member (__main__.CommTest)
2022-05-18T04:16:48.9585382Z test_nccl_warn_not_in_group_debug_detail (__main__.CommTest)
2022-05-18T04:16:48.9585642Z test_nccl_warn_not_in_group_debug_info (__main__.CommTest)
2022-05-18T04:16:48.9585895Z test_nccl_warn_not_in_group_debug_off (__main__.CommTest)
2022-05-18T04:16:48.9586151Z test_pass_nccl_options_high_priority_stream (__main__.CommTest)
2022-05-18T04:16:48.9586424Z test_sequence_num_incremented_nccl_default (__main__.CommTest)
2022-05-18T04:16:48.9586695Z test_sequence_num_incremented_nccl_subgroup (__main__.CommTest)
2022-05-18T04:16:48.9586947Z test_sequence_num_set_default_pg_nccl (__main__.CommTest)
2022-05-18T04:16:48.9587199Z test_sequence_num_set_nccl_new_group (__main__.CommTest)
2022-05-18T04:16:48.9594368Z <unittest.suite.TestSuite tests=[<__main__.DistributedDataParallelTest testMethod=test_accumulate_gradients_module>, <__main__.DistributedDataParallelTest testMethod=test_accumulate_gradients_module_with_grad_is_view>, <__main__.DistributedDataParallelTest testMethod=test_arbitrary_forward_return_value>, <__main__.DistributedDataParallelTest testMethod=test_arbitrary_forward_return_value_grad_is_view>, <__main__.DistributedDataParallelTest testMethod=test_bf16_compress_wrapper_is_view>, <__main__.DistributedDataParallelTest testMethod=test_bf16_compress_wrapper_nccl>, <__main__.DistributedDataParallelTest testMethod=test_builtin_ddp_comm_hooks_nccl>, <__main__.DistributedDataParallelTest testMethod=test_builtin_ddp_comm_hooks_nccl_grad_is_view>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_dynamic_module>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_dynamic_weight_sharing>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_once_use_reentrant_False>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_once_use_reentrant_True>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_twice_static_graph_use_reentrant_False>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_twice_static_graph_use_reentrant_True>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_twice_use_reentrant_False>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_twice_use_reentrant_True>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_twice_weight_sharing>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_unused_params_use_reentrant_False>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_unused_params_use_reentrant_True>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_weight_sharing_use_reentrant_False>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_weight_sharing_use_reentrant_True>, <__main__.DistributedDataParallelTest testMethod=test_ddp_comm_hook_allreduce_hook_nccl>, <__main__.DistributedDataParallelTest testMethod=test_ddp_comm_hook_allreduce_hook_nccl_grad_is_view>, <__main__.DistributedDataParallelTest testMethod=test_ddp_comm_hook_allreduce_hook_nccl_static_graph>, <__main__.DistributedDataParallelTest testMethod=test_ddp_comm_hook_allreduce_with_then_hook_nccl>, <__main__.DistributedDataParallelTest testMethod=test_ddp_comm_hook_future_passing_gpu_nccl>, <__main__.DistributedDataParallelTest testMethod=test_ddp_multi_device_module_config>, <__main__.DistributedDataParallelTest testMethod=test_ddp_weight_sharing>, <__main__.DistributedDataParallelTest testMethod=test_ddp_with_lazy_parameters>, <__main__.DistributedDataParallelTest testMethod=test_default_ddp_comm_hooks_nccl>, <__main__.DistributedDataParallelTest testMethod=test_default_ddp_comm_hooks_nccl_is_view>, <__main__.DistributedDataParallelTest testMethod=test_failure_recovery>, <__main__.DistributedDataParallelTest testMethod=test_find_unused_parameters_kwarg_debug_detail>, <__main__.DistributedDataParallelTest testMethod=test_find_unused_parameters_kwarg_debug_info>, <__main__.DistributedDataParallelTest testMethod=test_find_unused_parameters_kwarg_debug_off>, <__main__.DistributedDataParallelTest testMethod=test_find_unused_parameters_kwarg_grad_is_view_debug_detail>, <__main__.DistributedDataParallelTest testMethod=test_find_unused_parameters_kwarg_grad_is_view_debug_info>, <__main__.DistributedDataParallelTest testMethod=test_find_unused_parameters_kwarg_grad_is_view_debug_off>, <__main__.DistributedDataParallelTest testMethod=test_fp16>, <__main__.DistributedDataParallelTest testMethod=test_fp16_compress_wrapper_is_view>, <__main__.DistributedDataParallelTest testMethod=test_fp16_compress_wrapper_nccl>, <__main__.DistributedDataParallelTest testMethod=test_fp16_grad_is_view>, <__main__.DistributedDataParallelTest testMethod=test_grad_layout_1devicemodule_1replicaperprocess>, <__main__.DistributedDataParallelTest testMethod=test_grad_layout_2devicemodule>, <__main__.DistributedDataParallelTest testMethod=test_invalid_powerSGD_state>, <__main__.DistributedDataParallelTest testMethod=test_multiple_outputs_multiple_backward>, <__main__.DistributedDataParallelTest testMethod=test_multiple_outputs_multiple_backward_grad_is_view>, <__main__.DistributedDataParallelTest testMethod=test_nccl_backend_1gpu_module_device_ids_integer_list>, <__main__.DistributedDataParallelTest testMethod=test_nccl_backend_1gpu_module_device_ids_torch_device_list>, <__main__.DistributedDataParallelTest testMethod=test_nccl_backend_2gpu_module>, <__main__.DistributedDataParallelTest testMethod=test_nccl_backend_4gpu_module>, <__main__.DistributedDataParallelTest testMethod=test_nccl_backend_multi_device_ids_not_allowed>, <__main__.DistributedDataParallelTest testMethod=test_nccl_backend_multi_device_module_device_ids_None>, <__main__.DistributedDataParallelTest testMethod=test_nccl_backend_single_device_module_device_ids_None>, <__main__.DistributedDataParallelTest testMethod=test_nccl_backend_single_device_module_empty_device_ids>, <__main__.DistributedDataParallelTest testMethod=test_nccl_propagate_error_reason>, <__main__.DistributedDataParallelTest testMethod=test_no_grad>, <__main__.DistributedDataParallelTest testMethod=test_param_layout_mismatch_error>, <__main__.DistributedDataParallelTest testMethod=test_pass_default_pg>, <__main__.DistributedDataParallelTest testMethod=test_powerSGD_ddp_comm_hook_nccl>, <__main__.DistributedDataParallelTest testMethod=test_powerSGD_ddp_comm_hook_nccl_grad_is_view>, <__main__.DistributedDataParallelTest testMethod=test_sync_batch_norm_empty_input>, <__main__.DistributedDataParallelTest testMethod=test_sync_batch_norm_only_empty_input>]>
2022-05-18T04:16:48.9601347Z test_accumulate_gradients_module (__main__.DistributedDataParallelTest)
2022-05-18T04:16:48.9601686Z test_accumulate_gradients_module_with_grad_is_view (__main__.DistributedDataParallelTest)
2022-05-18T04:16:48.9602033Z test_arbitrary_forward_return_value (__main__.DistributedDataParallelTest)
2022-05-18T04:16:48.9602380Z test_arbitrary_forward_return_value_grad_is_view (__main__.DistributedDataParallelTest)
2022-05-18T04:16:48.9602719Z test_bf16_compress_wrapper_is_view (__main__.DistributedDataParallelTest)
2022-05-18T04:16:48.9603081Z test_bf16_compress_wrapper_nccl (__main__.DistributedDataParallelTest)
2022-05-18T04:16:48.9603405Z test_builtin_ddp_comm_hooks_nccl (__main__.DistributedDataParallelTest)
2022-05-18T04:16:48.9603781Z test_builtin_ddp_comm_hooks_nccl_grad_is_view (__main__.DistributedDataParallelTest)
2022-05-18T04:16:48.9604113Z test_ddp_checkpointing_dynamic_module (__main__.DistributedDataParallelTest)
2022-05-18T04:16:48.9604500Z test_ddp_checkpointing_dynamic_weight_sharing (__main__.DistributedDataParallelTest)
2022-05-18T04:16:48.9604863Z test_ddp_checkpointing_once_use_reentrant_False (__main__.DistributedDataParallelTest)
2022-05-18T04:16:48.9605217Z test_ddp_checkpointing_once_use_reentrant_True (__main__.DistributedDataParallelTest)
2022-05-18T04:16:48.9605578Z test_ddp_checkpointing_twice_static_graph_use_reentrant_False (__main__.DistributedDataParallelTest)
2022-05-18T04:16:48.9605963Z test_ddp_checkpointing_twice_static_graph_use_reentrant_True (__main__.DistributedDataParallelTest)
2022-05-18T04:16:48.9606342Z test_ddp_checkpointing_twice_use_reentrant_False (__main__.DistributedDataParallelTest)
2022-05-18T04:16:48.9606684Z test_ddp_checkpointing_twice_use_reentrant_True (__main__.DistributedDataParallelTest)
2022-05-18T04:16:48.9607037Z test_ddp_checkpointing_twice_weight_sharing (__main__.DistributedDataParallelTest)
2022-05-18T04:16:48.9607405Z test_ddp_checkpointing_unused_params_use_reentrant_False (__main__.DistributedDataParallelTest)
2022-05-18T04:16:48.9607778Z test_ddp_checkpointing_unused_params_use_reentrant_True (__main__.DistributedDataParallelTest)
2022-05-18T04:16:48.9608140Z test_ddp_checkpointing_weight_sharing_use_reentrant_False (__main__.DistributedDataParallelTest)
2022-05-18T04:16:48.9608518Z test_ddp_checkpointing_weight_sharing_use_reentrant_True (__main__.DistributedDataParallelTest)
2022-05-18T04:16:48.9608871Z test_ddp_comm_hook_allreduce_hook_nccl (__main__.DistributedDataParallelTest)
2022-05-18T04:16:48.9609200Z test_ddp_comm_hook_allreduce_hook_nccl_grad_is_view (__main__.DistributedDataParallelTest)
2022-05-18T04:16:48.9609556Z test_ddp_comm_hook_allreduce_hook_nccl_static_graph (__main__.DistributedDataParallelTest)
2022-05-18T04:16:48.9609910Z test_ddp_comm_hook_allreduce_with_then_hook_nccl (__main__.DistributedDataParallelTest)
2022-05-18T04:16:48.9610257Z test_ddp_comm_hook_future_passing_gpu_nccl (__main__.DistributedDataParallelTest)
2022-05-18T04:16:48.9610577Z test_ddp_multi_device_module_config (__main__.DistributedDataParallelTest)
2022-05-18T04:16:48.9610893Z test_ddp_weight_sharing (__main__.DistributedDataParallelTest)
2022-05-18T04:16:48.9611205Z test_ddp_with_lazy_parameters (__main__.DistributedDataParallelTest)
2022-05-18T04:16:48.9611512Z test_default_ddp_comm_hooks_nccl (__main__.DistributedDataParallelTest)
2022-05-18T04:16:48.9611834Z test_default_ddp_comm_hooks_nccl_is_view (__main__.DistributedDataParallelTest)
2022-05-18T04:16:48.9612148Z test_failure_recovery (__main__.DistributedDataParallelTest)
2022-05-18T04:16:48.9612473Z test_find_unused_parameters_kwarg_debug_detail (__main__.DistributedDataParallelTest)
2022-05-18T04:16:48.9612813Z test_find_unused_parameters_kwarg_debug_info (__main__.DistributedDataParallelTest)
2022-05-18T04:16:48.9613162Z test_find_unused_parameters_kwarg_debug_off (__main__.DistributedDataParallelTest)
2022-05-18T04:16:48.9613524Z test_find_unused_parameters_kwarg_grad_is_view_debug_detail (__main__.DistributedDataParallelTest)
2022-05-18T04:16:48.9613890Z test_find_unused_parameters_kwarg_grad_is_view_debug_info (__main__.DistributedDataParallelTest)
2022-05-18T04:16:48.9614264Z test_find_unused_parameters_kwarg_grad_is_view_debug_off (__main__.DistributedDataParallelTest)
2022-05-18T04:16:48.9614577Z test_fp16 (__main__.DistributedDataParallelTest)
2022-05-18T04:16:48.9614870Z test_fp16_compress_wrapper_is_view (__main__.DistributedDataParallelTest)
2022-05-18T04:16:48.9615179Z test_fp16_compress_wrapper_nccl (__main__.DistributedDataParallelTest)
2022-05-18T04:16:48.9615484Z test_fp16_grad_is_view (__main__.DistributedDataParallelTest)
2022-05-18T04:16:48.9615816Z test_grad_layout_1devicemodule_1replicaperprocess (__main__.DistributedDataParallelTest)
2022-05-18T04:16:48.9616152Z test_grad_layout_2devicemodule (__main__.DistributedDataParallelTest)
2022-05-18T04:16:48.9616514Z test_invalid_powerSGD_state (__main__.DistributedDataParallelTest)
2022-05-18T04:16:48.9616897Z test_multiple_outputs_multiple_backward (__main__.DistributedDataParallelTest)
2022-05-18T04:16:48.9617253Z test_multiple_outputs_multiple_backward_grad_is_view (__main__.DistributedDataParallelTest)
2022-05-18T04:16:48.9617602Z test_nccl_backend_1gpu_module_device_ids_integer_list (__main__.DistributedDataParallelTest)
2022-05-18T04:16:48.9617971Z test_nccl_backend_1gpu_module_device_ids_torch_device_list (__main__.DistributedDataParallelTest)
2022-05-18T04:16:48.9618315Z test_nccl_backend_2gpu_module (__main__.DistributedDataParallelTest)
2022-05-18T04:16:48.9618618Z test_nccl_backend_4gpu_module (__main__.DistributedDataParallelTest)
2022-05-18T04:16:48.9618949Z test_nccl_backend_multi_device_ids_not_allowed (__main__.DistributedDataParallelTest)
2022-05-18T04:16:48.9619305Z test_nccl_backend_multi_device_module_device_ids_None (__main__.DistributedDataParallelTest)
2022-05-18T04:16:48.9619672Z test_nccl_backend_single_device_module_device_ids_None (__main__.DistributedDataParallelTest)
2022-05-18T04:16:48.9620026Z test_nccl_backend_single_device_module_empty_device_ids (__main__.DistributedDataParallelTest)
2022-05-18T04:16:48.9620369Z test_nccl_propagate_error_reason (__main__.DistributedDataParallelTest)
2022-05-18T04:16:48.9620668Z test_no_grad (__main__.DistributedDataParallelTest)
2022-05-18T04:16:48.9620951Z test_param_layout_mismatch_error (__main__.DistributedDataParallelTest)
2022-05-18T04:16:48.9621259Z test_pass_default_pg (__main__.DistributedDataParallelTest)
2022-05-18T04:16:48.9621563Z test_powerSGD_ddp_comm_hook_nccl (__main__.DistributedDataParallelTest)
2022-05-18T04:16:48.9621896Z test_powerSGD_ddp_comm_hook_nccl_grad_is_view (__main__.DistributedDataParallelTest)
2022-05-18T04:16:48.9622213Z test_sync_batch_norm_empty_input (__main__.DistributedDataParallelTest)
2022-05-18T04:16:48.9622536Z test_sync_batch_norm_only_empty_input (__main__.DistributedDataParallelTest)
2022-05-18T04:16:48.9622821Z <unittest.suite.TestSuite tests=[]>
2022-05-18T04:16:48.9623974Z <unittest.suite.TestSuite tests=[<__main__.NcclErrorHandlingTest testMethod=test_invalid_nccl_blocking_wait_env>, <__main__.NcclErrorHandlingTest testMethod=test_nccl_blocking_wait_with_barrier>, <__main__.NcclErrorHandlingTest testMethod=test_nccl_errors_blocking_abort>, <__main__.NcclErrorHandlingTest testMethod=test_nccl_errors_blocking_clean_exit>, <__main__.NcclErrorHandlingTest testMethod=test_nccl_errors_blocking_nonzero_exit>, <__main__.NcclErrorHandlingTest testMethod=test_nccl_errors_blocking_sigkill>, <__main__.NcclErrorHandlingTest testMethod=test_nccl_errors_blocking_sigterm>, <__main__.NcclErrorHandlingTest testMethod=test_nccl_errors_nonblocking>, <__main__.NcclErrorHandlingTest testMethod=test_nccl_timeout>]>
2022-05-18T04:16:48.9624927Z test_invalid_nccl_blocking_wait_env (__main__.NcclErrorHandlingTest)
2022-05-18T04:16:48.9625238Z test_nccl_blocking_wait_with_barrier (__main__.NcclErrorHandlingTest)
2022-05-18T04:16:48.9625525Z test_nccl_errors_blocking_abort (__main__.NcclErrorHandlingTest)
2022-05-18T04:16:48.9625821Z test_nccl_errors_blocking_clean_exit (__main__.NcclErrorHandlingTest)
2022-05-18T04:16:48.9626122Z test_nccl_errors_blocking_nonzero_exit (__main__.NcclErrorHandlingTest)
2022-05-18T04:16:48.9626408Z test_nccl_errors_blocking_sigkill (__main__.NcclErrorHandlingTest)
2022-05-18T04:16:48.9626702Z test_nccl_errors_blocking_sigterm (__main__.NcclErrorHandlingTest)
2022-05-18T04:16:48.9626996Z test_nccl_errors_nonblocking (__main__.NcclErrorHandlingTest)
2022-05-18T04:16:48.9627254Z test_nccl_timeout (__main__.NcclErrorHandlingTest)
2022-05-18T04:16:48.9627596Z <unittest.suite.TestSuite tests=[<__main__.ProcessGroupNCCLNoGPUTest testMethod=test_init_no_gpus>]>
2022-05-18T04:16:48.9627937Z test_init_no_gpus (__main__.ProcessGroupNCCLNoGPUTest)
2022-05-18T04:16:48.9629432Z <unittest.suite.TestSuite tests=[<__main__.ProcessGroupNCCLTest testMethod=test_allgather_base_basics>, <__main__.ProcessGroupNCCLTest testMethod=test_allgather_base_ops>, <__main__.ProcessGroupNCCLTest testMethod=test_allgather_ops>, <__main__.ProcessGroupNCCLTest testMethod=test_allreduce_ops>, <__main__.ProcessGroupNCCLTest testMethod=test_barrier>, <__main__.ProcessGroupNCCLTest testMethod=test_broadcast_ops>, <__main__.ProcessGroupNCCLTest testMethod=test_empty_tensors>, <__main__.ProcessGroupNCCLTest testMethod=test_gather_checks>, <__main__.ProcessGroupNCCLTest testMethod=test_gather_ops>, <__main__.ProcessGroupNCCLTest testMethod=test_gather_stress>, <__main__.ProcessGroupNCCLTest testMethod=test_reduce_ops>, <__main__.ProcessGroupNCCLTest testMethod=test_reduce_scatter_base_basics>, <__main__.ProcessGroupNCCLTest testMethod=test_reduce_scatter_base_ops>, <__main__.ProcessGroupNCCLTest testMethod=test_reduce_scatter_ops>, <__main__.ProcessGroupNCCLTest testMethod=test_scatter_checks>, <__main__.ProcessGroupNCCLTest testMethod=test_scatter_ops>, <__main__.ProcessGroupNCCLTest testMethod=test_scatter_stress>]>
2022-05-18T04:16:48.9630964Z test_allgather_base_basics (__main__.ProcessGroupNCCLTest)
2022-05-18T04:16:48.9631247Z test_allgather_base_ops (__main__.ProcessGroupNCCLTest)
2022-05-18T04:16:48.9631519Z test_allgather_ops (__main__.ProcessGroupNCCLTest)
2022-05-18T04:16:48.9631769Z test_allreduce_ops (__main__.ProcessGroupNCCLTest)
2022-05-18T04:16:48.9632022Z test_barrier (__main__.ProcessGroupNCCLTest)
2022-05-18T04:16:48.9632278Z test_broadcast_ops (__main__.ProcessGroupNCCLTest)
2022-05-18T04:16:48.9632539Z test_empty_tensors (__main__.ProcessGroupNCCLTest)
2022-05-18T04:16:48.9632782Z test_gather_checks (__main__.ProcessGroupNCCLTest)
2022-05-18T04:16:48.9633036Z test_gather_ops (__main__.ProcessGroupNCCLTest)
2022-05-18T04:16:48.9633291Z test_gather_stress (__main__.ProcessGroupNCCLTest)
2022-05-18T04:16:48.9633532Z test_reduce_ops (__main__.ProcessGroupNCCLTest)
2022-05-18T04:16:48.9633809Z test_reduce_scatter_base_basics (__main__.ProcessGroupNCCLTest)
2022-05-18T04:16:48.9634099Z test_reduce_scatter_base_ops (__main__.ProcessGroupNCCLTest)
2022-05-18T04:16:48.9634364Z test_reduce_scatter_ops (__main__.ProcessGroupNCCLTest)
2022-05-18T04:16:48.9634634Z test_scatter_checks (__main__.ProcessGroupNCCLTest)
2022-05-18T04:16:48.9634893Z test_scatter_ops (__main__.ProcessGroupNCCLTest)
2022-05-18T04:16:48.9635138Z test_scatter_stress (__main__.ProcessGroupNCCLTest)
2022-05-18T04:16:48.9635457Z <unittest.suite.TestSuite tests=[<__main__.RendezvousEnvTest testMethod=test_common_errors>]>
2022-05-18T04:16:48.9635760Z test_common_errors (__main__.RendezvousEnvTest)
2022-05-18T04:16:48.9635999Z <unittest.suite.TestSuite tests=[]>
2022-05-18T04:16:48.9636295Z <unittest.suite.TestSuite tests=[<__main__.TimeoutTest testMethod=test_default_store_timeout_nccl>]>
2022-05-18T04:16:48.9636609Z test_default_store_timeout_nccl (__main__.TimeoutTest)
2022-05-18T04:16:49.5279517Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:16:49.5289432Z 
2022-05-18T04:16:49.5289582Z Running tests...
2022-05-18T04:16:49.5290171Z ----------------------------------------------------------------------
2022-05-18T04:16:49.5298617Z   test_all_reduce_coalesced_nccl (__main__.CommTest) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-05-18T04:16:49.5298978Z 
2022-05-18T04:16:49.5299252Z ----------------------------------------------------------------------
2022-05-18T04:16:49.5299502Z Ran 1 test in 0.001s
2022-05-18T04:16:49.5299615Z 
2022-05-18T04:16:49.5299674Z OK (skipped=1)
2022-05-18T04:16:49.5299784Z 
2022-05-18T04:16:49.5299870Z Generating XML reports...
2022-05-18T04:16:49.5323746Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220518041649.xml
2022-05-18T04:16:50.1866994Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:16:50.1877091Z 
2022-05-18T04:16:50.1877214Z Running tests...
2022-05-18T04:16:50.1878104Z ----------------------------------------------------------------------
2022-05-18T04:16:50.1883760Z   test_broadcast_coalesced_nccl (__main__.CommTest) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-05-18T04:16:50.1885446Z 
2022-05-18T04:16:50.1886038Z ----------------------------------------------------------------------
2022-05-18T04:16:50.1886467Z Ran 1 test in 0.001s
2022-05-18T04:16:50.1886663Z 
2022-05-18T04:16:50.1886790Z OK (skipped=1)
2022-05-18T04:16:50.1886950Z 
2022-05-18T04:16:50.1887090Z Generating XML reports...
2022-05-18T04:16:50.1910364Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220518041650.xml
2022-05-18T04:16:50.8580217Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:16:50.8589905Z 
2022-05-18T04:16:50.8590045Z Running tests...
2022-05-18T04:16:50.8590460Z ----------------------------------------------------------------------
2022-05-18T04:16:50.8606869Z   test_nccl_barrier (__main__.CommTest) ... skip: c10d was not compiled with the NCCL backend (0.002s)
2022-05-18T04:16:50.8607219Z 
2022-05-18T04:16:50.8607672Z ----------------------------------------------------------------------
2022-05-18T04:16:50.8608123Z Ran 1 test in 0.002s
2022-05-18T04:16:50.8608316Z 
2022-05-18T04:16:50.8608451Z OK (skipped=1)
2022-05-18T04:16:50.8608648Z 
2022-05-18T04:16:50.8608794Z Generating XML reports...
2022-05-18T04:16:50.8639710Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220518041650.xml
2022-05-18T04:16:51.5313045Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:16:51.5323413Z 
2022-05-18T04:16:51.5323528Z Running tests...
2022-05-18T04:16:51.5324119Z ----------------------------------------------------------------------
2022-05-18T04:16:51.5329436Z   test_nccl_barrier_device_ids (__main__.CommTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-05-18T04:16:51.5329833Z 
2022-05-18T04:16:51.5330312Z ----------------------------------------------------------------------
2022-05-18T04:16:51.5330597Z Ran 1 test in 0.001s
2022-05-18T04:16:51.5330711Z 
2022-05-18T04:16:51.5330786Z OK (skipped=1)
2022-05-18T04:16:51.5330898Z 
2022-05-18T04:16:51.5330985Z Generating XML reports...
2022-05-18T04:16:51.5354785Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220518041651.xml
2022-05-18T04:16:52.2043199Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:16:52.2053236Z 
2022-05-18T04:16:52.2053529Z Running tests...
2022-05-18T04:16:52.2054179Z ----------------------------------------------------------------------
2022-05-18T04:16:52.2059019Z   test_nccl_barrier_device_ids_function_argument (__main__.CommTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-05-18T04:16:52.2059389Z 
2022-05-18T04:16:52.2059776Z ----------------------------------------------------------------------
2022-05-18T04:16:52.2060249Z Ran 1 test in 0.001s
2022-05-18T04:16:52.2060391Z 
2022-05-18T04:16:52.2060464Z OK (skipped=1)
2022-05-18T04:16:52.2060572Z 
2022-05-18T04:16:52.2060658Z Generating XML reports...
2022-05-18T04:16:52.2085735Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220518041652.xml
2022-05-18T04:16:52.8813228Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:16:52.8822252Z 
2022-05-18T04:16:52.8822331Z Running tests...
2022-05-18T04:16:52.8823556Z ----------------------------------------------------------------------
2022-05-18T04:16:52.8830042Z   test_nccl_barrier_timeout (__main__.CommTest) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-05-18T04:16:52.8830379Z 
2022-05-18T04:16:52.8830739Z ----------------------------------------------------------------------
2022-05-18T04:16:52.8831193Z Ran 1 test in 0.001s
2022-05-18T04:16:52.8831398Z 
2022-05-18T04:16:52.8831476Z OK (skipped=1)
2022-05-18T04:16:52.8831786Z 
2022-05-18T04:16:52.8831877Z Generating XML reports...
2022-05-18T04:16:52.8861800Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220518041652.xml
2022-05-18T04:16:53.5487403Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:16:53.5496786Z 
2022-05-18T04:16:53.5496888Z Running tests...
2022-05-18T04:16:53.5497466Z ----------------------------------------------------------------------
2022-05-18T04:16:53.5506395Z   test_nccl_barrier_timeout_new_group (__main__.CommTest) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-05-18T04:16:53.5506939Z 
2022-05-18T04:16:53.5507273Z ----------------------------------------------------------------------
2022-05-18T04:16:53.5507723Z Ran 1 test in 0.001s
2022-05-18T04:16:53.5507937Z 
2022-05-18T04:16:53.5508060Z OK (skipped=1)
2022-05-18T04:16:53.5508182Z 
2022-05-18T04:16:53.5508267Z Generating XML reports...
2022-05-18T04:16:53.5531691Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220518041653.xml
2022-05-18T04:16:54.2194080Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:16:54.2204399Z 
2022-05-18T04:16:54.2204671Z Running tests...
2022-05-18T04:16:54.2205308Z ----------------------------------------------------------------------
2022-05-18T04:16:54.2214317Z   test_nccl_barrier_timeout_new_group_non_member (__main__.CommTest) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-05-18T04:16:54.2214732Z 
2022-05-18T04:16:54.2215118Z ----------------------------------------------------------------------
2022-05-18T04:16:54.2215502Z Ran 1 test in 0.001s
2022-05-18T04:16:54.2215691Z 
2022-05-18T04:16:54.2215816Z OK (skipped=1)
2022-05-18T04:16:54.2215997Z 
2022-05-18T04:16:54.2216139Z Generating XML reports...
2022-05-18T04:16:54.2240425Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220518041654.xml
2022-05-18T04:16:54.8930952Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:16:54.8939756Z 
2022-05-18T04:16:54.8939850Z Running tests...
2022-05-18T04:16:54.8940409Z ----------------------------------------------------------------------
2022-05-18T04:16:54.8944483Z   test_nccl_warn_not_in_group_debug_detail (__main__.CommTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-05-18T04:16:54.8944802Z 
2022-05-18T04:16:54.8945201Z ----------------------------------------------------------------------
2022-05-18T04:16:54.8945631Z Ran 1 test in 0.001s
2022-05-18T04:16:54.8945841Z 
2022-05-18T04:16:54.8945941Z OK (skipped=1)
2022-05-18T04:16:54.8946066Z 
2022-05-18T04:16:54.8946152Z Generating XML reports...
2022-05-18T04:16:54.8983759Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220518041654.xml
2022-05-18T04:16:55.5667246Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:16:55.5676295Z 
2022-05-18T04:16:55.5676780Z Running tests...
2022-05-18T04:16:55.5677250Z ----------------------------------------------------------------------
2022-05-18T04:16:55.5681118Z   test_nccl_warn_not_in_group_debug_info (__main__.CommTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-05-18T04:16:55.5681512Z 
2022-05-18T04:16:55.5681906Z ----------------------------------------------------------------------
2022-05-18T04:16:55.5682313Z Ran 1 test in 0.000s
2022-05-18T04:16:55.5682489Z 
2022-05-18T04:16:55.5682609Z OK (skipped=1)
2022-05-18T04:16:55.5682787Z 
2022-05-18T04:16:55.5682928Z Generating XML reports...
2022-05-18T04:16:55.5707223Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220518041655.xml
2022-05-18T04:16:56.2353986Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:16:56.2363500Z 
2022-05-18T04:16:56.2363950Z Running tests...
2022-05-18T04:16:56.2364528Z ----------------------------------------------------------------------
2022-05-18T04:16:56.2368853Z   test_nccl_warn_not_in_group_debug_off (__main__.CommTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-05-18T04:16:56.2369300Z 
2022-05-18T04:16:56.2369581Z ----------------------------------------------------------------------
2022-05-18T04:16:56.2369832Z Ran 1 test in 0.000s
2022-05-18T04:16:56.2369933Z 
2022-05-18T04:16:56.2370008Z OK (skipped=1)
2022-05-18T04:16:56.2370115Z 
2022-05-18T04:16:56.2370202Z Generating XML reports...
2022-05-18T04:16:56.2393422Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220518041656.xml
2022-05-18T04:16:56.9053115Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:16:56.9062651Z 
2022-05-18T04:16:56.9062759Z Running tests...
2022-05-18T04:16:56.9063377Z ----------------------------------------------------------------------
2022-05-18T04:16:56.9072220Z   test_pass_nccl_options_high_priority_stream (__main__.CommTest) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-05-18T04:16:56.9072762Z 
2022-05-18T04:16:56.9073674Z ----------------------------------------------------------------------
2022-05-18T04:16:56.9074069Z Ran 1 test in 0.001s
2022-05-18T04:16:56.9074186Z 
2022-05-18T04:16:56.9074267Z OK (skipped=1)
2022-05-18T04:16:56.9074378Z 
2022-05-18T04:16:56.9074471Z Generating XML reports...
2022-05-18T04:16:56.9105040Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220518041656.xml
2022-05-18T04:16:57.5660755Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:16:57.5670871Z 
2022-05-18T04:16:57.5670949Z Running tests...
2022-05-18T04:16:57.5671491Z ----------------------------------------------------------------------
2022-05-18T04:16:57.5675641Z   test_sequence_num_incremented_nccl_default (__main__.CommTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-05-18T04:16:57.5676194Z 
2022-05-18T04:16:57.5676920Z ----------------------------------------------------------------------
2022-05-18T04:16:57.5677343Z Ran 1 test in 0.000s
2022-05-18T04:16:57.5677470Z 
2022-05-18T04:16:57.5677549Z OK (skipped=1)
2022-05-18T04:16:57.5677658Z 
2022-05-18T04:16:57.5677745Z Generating XML reports...
2022-05-18T04:16:57.5700927Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220518041657.xml
2022-05-18T04:16:58.2365830Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:16:58.2375633Z 
2022-05-18T04:16:58.2375747Z Running tests...
2022-05-18T04:16:58.2376252Z ----------------------------------------------------------------------
2022-05-18T04:16:58.2381908Z   test_sequence_num_incremented_nccl_subgroup (__main__.CommTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-05-18T04:16:58.2382202Z 
2022-05-18T04:16:58.2382470Z ----------------------------------------------------------------------
2022-05-18T04:16:58.2382752Z Ran 1 test in 0.000s
2022-05-18T04:16:58.2383042Z 
2022-05-18T04:16:58.2383127Z OK (skipped=1)
2022-05-18T04:16:58.2383236Z 
2022-05-18T04:16:58.2383323Z Generating XML reports...
2022-05-18T04:16:58.2405533Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220518041658.xml
2022-05-18T04:16:58.9207595Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:16:58.9218390Z 
2022-05-18T04:16:58.9218602Z Running tests...
2022-05-18T04:16:58.9219215Z ----------------------------------------------------------------------
2022-05-18T04:16:58.9223758Z   test_sequence_num_set_default_pg_nccl (__main__.CommTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-05-18T04:16:58.9224167Z 
2022-05-18T04:16:58.9224635Z ----------------------------------------------------------------------
2022-05-18T04:16:58.9225333Z Ran 1 test in 0.000s
2022-05-18T04:16:58.9225541Z 
2022-05-18T04:16:58.9225683Z OK (skipped=1)
2022-05-18T04:16:58.9225878Z 
2022-05-18T04:16:58.9226032Z Generating XML reports...
2022-05-18T04:16:58.9255748Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220518041658.xml
2022-05-18T04:16:59.6096108Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:16:59.6106438Z 
2022-05-18T04:16:59.6106555Z Running tests...
2022-05-18T04:16:59.6107139Z ----------------------------------------------------------------------
2022-05-18T04:16:59.6112049Z   test_sequence_num_set_nccl_new_group (__main__.CommTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-05-18T04:16:59.6112303Z 
2022-05-18T04:16:59.6112573Z ----------------------------------------------------------------------
2022-05-18T04:16:59.6112825Z Ran 1 test in 0.000s
2022-05-18T04:16:59.6112941Z 
2022-05-18T04:16:59.6113027Z OK (skipped=1)
2022-05-18T04:16:59.6113122Z 
2022-05-18T04:16:59.6113210Z Generating XML reports...
2022-05-18T04:16:59.6137416Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220518041659.xml
2022-05-18T04:17:00.2948824Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:17:00.2959171Z 
2022-05-18T04:17:00.2959610Z Running tests...
2022-05-18T04:17:00.2960006Z ----------------------------------------------------------------------
2022-05-18T04:17:00.2963689Z   test_accumulate_gradients_module (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-05-18T04:17:00.2964040Z 
2022-05-18T04:17:00.2964295Z ----------------------------------------------------------------------
2022-05-18T04:17:00.2964541Z Ran 1 test in 0.000s
2022-05-18T04:17:00.2964658Z 
2022-05-18T04:17:00.2964722Z OK (skipped=1)
2022-05-18T04:17:00.2964831Z 
2022-05-18T04:17:00.2964932Z Generating XML reports...
2022-05-18T04:17:00.2989217Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518041700.xml
2022-05-18T04:17:00.9745118Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:17:00.9785864Z 
2022-05-18T04:17:00.9786164Z Running tests...
2022-05-18T04:17:00.9786904Z ----------------------------------------------------------------------
2022-05-18T04:17:00.9787529Z   test_accumulate_gradients_module_with_grad_is_view (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-05-18T04:17:00.9787912Z 
2022-05-18T04:17:00.9788199Z ----------------------------------------------------------------------
2022-05-18T04:17:00.9788556Z Ran 1 test in 0.000s
2022-05-18T04:17:00.9788730Z 
2022-05-18T04:17:00.9788843Z OK (skipped=1)
2022-05-18T04:17:00.9789017Z 
2022-05-18T04:17:00.9789147Z Generating XML reports...
2022-05-18T04:17:00.9801102Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518041700.xml
2022-05-18T04:17:01.6590877Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:17:01.6600920Z 
2022-05-18T04:17:01.6601018Z Running tests...
2022-05-18T04:17:01.6601482Z ----------------------------------------------------------------------
2022-05-18T04:17:01.6605927Z   test_arbitrary_forward_return_value (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-05-18T04:17:01.6606506Z 
2022-05-18T04:17:01.6606900Z ----------------------------------------------------------------------
2022-05-18T04:17:01.6607272Z Ran 1 test in 0.000s
2022-05-18T04:17:01.6607444Z 
2022-05-18T04:17:01.6607538Z OK (skipped=1)
2022-05-18T04:17:01.6607684Z 
2022-05-18T04:17:01.6607810Z Generating XML reports...
2022-05-18T04:17:01.6632659Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518041701.xml
2022-05-18T04:17:02.3649541Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:17:02.3659415Z 
2022-05-18T04:17:02.3659963Z Running tests...
2022-05-18T04:17:02.3660519Z ----------------------------------------------------------------------
2022-05-18T04:17:02.3664329Z   test_arbitrary_forward_return_value_grad_is_view (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-05-18T04:17:02.3664828Z 
2022-05-18T04:17:02.3665344Z ----------------------------------------------------------------------
2022-05-18T04:17:02.3665680Z Ran 1 test in 0.000s
2022-05-18T04:17:02.3665795Z 
2022-05-18T04:17:02.3665868Z OK (skipped=1)
2022-05-18T04:17:02.3665977Z 
2022-05-18T04:17:02.3666063Z Generating XML reports...
2022-05-18T04:17:02.3691611Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518041702.xml
2022-05-18T04:17:03.0473281Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:17:03.0482565Z 
2022-05-18T04:17:03.0482708Z Running tests...
2022-05-18T04:17:03.0483397Z ----------------------------------------------------------------------
2022-05-18T04:17:03.0488166Z   test_bf16_compress_wrapper_is_view (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-05-18T04:17:03.0488600Z 
2022-05-18T04:17:03.0489065Z ----------------------------------------------------------------------
2022-05-18T04:17:03.0489450Z Ran 1 test in 0.001s
2022-05-18T04:17:03.0489563Z 
2022-05-18T04:17:03.0489635Z OK (skipped=1)
2022-05-18T04:17:03.0489730Z 
2022-05-18T04:17:03.0489815Z Generating XML reports...
2022-05-18T04:17:03.0514297Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518041703.xml
2022-05-18T04:17:03.7293957Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:17:03.7304711Z 
2022-05-18T04:17:03.7305076Z Running tests...
2022-05-18T04:17:03.7305479Z ----------------------------------------------------------------------
2022-05-18T04:17:03.7310057Z   test_bf16_compress_wrapper_nccl (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-05-18T04:17:03.7310464Z 
2022-05-18T04:17:03.7310770Z ----------------------------------------------------------------------
2022-05-18T04:17:03.7311019Z Ran 1 test in 0.001s
2022-05-18T04:17:03.7311120Z 
2022-05-18T04:17:03.7311193Z OK (skipped=1)
2022-05-18T04:17:03.7311300Z 
2022-05-18T04:17:03.7311387Z Generating XML reports...
2022-05-18T04:17:03.7346848Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518041703.xml
2022-05-18T04:17:04.4128577Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:17:04.4138137Z 
2022-05-18T04:17:04.4138246Z Running tests...
2022-05-18T04:17:04.4138831Z ----------------------------------------------------------------------
2022-05-18T04:17:04.4142625Z   test_builtin_ddp_comm_hooks_nccl (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-05-18T04:17:04.4143149Z 
2022-05-18T04:17:04.4143453Z ----------------------------------------------------------------------
2022-05-18T04:17:04.4143689Z Ran 1 test in 0.000s
2022-05-18T04:17:04.4143805Z 
2022-05-18T04:17:04.4143880Z OK (skipped=1)
2022-05-18T04:17:04.4143989Z 
2022-05-18T04:17:04.4144073Z Generating XML reports...
2022-05-18T04:17:04.4167694Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518041704.xml
2022-05-18T04:17:05.0845824Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:17:05.0855960Z 
2022-05-18T04:17:05.0856092Z Running tests...
2022-05-18T04:17:05.0857113Z ----------------------------------------------------------------------
2022-05-18T04:17:05.0860860Z   test_builtin_ddp_comm_hooks_nccl_grad_is_view (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-05-18T04:17:05.0861348Z 
2022-05-18T04:17:05.0861719Z ----------------------------------------------------------------------
2022-05-18T04:17:05.0862159Z Ran 1 test in 0.000s
2022-05-18T04:17:05.0862367Z 
2022-05-18T04:17:05.0862509Z OK (skipped=1)
2022-05-18T04:17:05.0862651Z 
2022-05-18T04:17:05.0862726Z Generating XML reports...
2022-05-18T04:17:05.0886271Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518041705.xml
2022-05-18T04:17:05.7556445Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:17:05.7566811Z 
2022-05-18T04:17:05.7567104Z Running tests...
2022-05-18T04:17:05.7567733Z ----------------------------------------------------------------------
2022-05-18T04:17:05.7574539Z   test_ddp_checkpointing_dynamic_module (__main__.DistributedDataParallelTest)
2022-05-18T04:17:06.0379422Z Dynamic module can be checkpointed, multiple times, with non-reentrant ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9033
2022-05-18T04:17:06.0401091Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9034
2022-05-18T04:17:06.6123074Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:17:06.6322444Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:17:06.7425047Z skip: Need at least 2 CUDA devices (0.986s)
2022-05-18T04:17:06.7425374Z 
2022-05-18T04:17:06.7425830Z ----------------------------------------------------------------------
2022-05-18T04:17:06.7426086Z Ran 1 test in 0.986s
2022-05-18T04:17:06.7426202Z 
2022-05-18T04:17:06.7426276Z OK (skipped=1)
2022-05-18T04:17:06.7426398Z 
2022-05-18T04:17:06.7426482Z Generating XML reports...
2022-05-18T04:17:06.7465862Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518041705.xml
2022-05-18T04:17:07.5075206Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:17:07.5084607Z 
2022-05-18T04:17:07.5084740Z Running tests...
2022-05-18T04:17:07.5085328Z ----------------------------------------------------------------------
2022-05-18T04:17:07.5092614Z   test_ddp_checkpointing_dynamic_weight_sharing (__main__.DistributedDataParallelTest)
2022-05-18T04:17:07.7915046Z Dynamic module can be checkpointed multiple times with weight sharing ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9066
2022-05-18T04:17:07.7937614Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9067
2022-05-18T04:17:08.3795826Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:17:08.4047681Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:17:08.5964435Z skip: Need at least 2 CUDA devices (1.087s)
2022-05-18T04:17:08.5964750Z 
2022-05-18T04:17:08.5965208Z ----------------------------------------------------------------------
2022-05-18T04:17:08.5965580Z Ran 1 test in 1.088s
2022-05-18T04:17:08.5965761Z 
2022-05-18T04:17:08.5965890Z OK (skipped=1)
2022-05-18T04:17:08.5966063Z 
2022-05-18T04:17:08.5966211Z Generating XML reports...
2022-05-18T04:17:08.6000321Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518041707.xml
2022-05-18T04:17:09.3641419Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:17:09.3650590Z 
2022-05-18T04:17:09.3650809Z Running tests...
2022-05-18T04:17:09.3651223Z ----------------------------------------------------------------------
2022-05-18T04:17:09.3659600Z   test_ddp_checkpointing_once_use_reentrant_False (__main__.DistributedDataParallelTest)
2022-05-18T04:17:09.6440607Z DDP works as expected when layer is checkpointed only once. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9099
2022-05-18T04:17:09.6462178Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9100
2022-05-18T04:17:10.2154862Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:17:10.2164000Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:17:10.3485474Z skip: Need at least 2 CUDA devices (0.983s)
2022-05-18T04:17:10.3485772Z 
2022-05-18T04:17:10.3486209Z ----------------------------------------------------------------------
2022-05-18T04:17:10.3486466Z Ran 1 test in 0.983s
2022-05-18T04:17:10.3486584Z 
2022-05-18T04:17:10.3486658Z OK (skipped=1)
2022-05-18T04:17:10.3486764Z 
2022-05-18T04:17:10.3486856Z Generating XML reports...
2022-05-18T04:17:10.3519630Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518041709.xml
2022-05-18T04:17:11.1072124Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:17:11.1081823Z 
2022-05-18T04:17:11.1081968Z Running tests...
2022-05-18T04:17:11.1082436Z ----------------------------------------------------------------------
2022-05-18T04:17:11.1090606Z   test_ddp_checkpointing_once_use_reentrant_True (__main__.DistributedDataParallelTest)
2022-05-18T04:17:11.3896182Z DDP works as expected when layer is checkpointed only once. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9132
2022-05-18T04:17:11.3918456Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9133
2022-05-18T04:17:11.9612369Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:17:11.9618992Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:17:12.0941985Z skip: Need at least 2 CUDA devices (0.986s)
2022-05-18T04:17:12.0942310Z 
2022-05-18T04:17:12.0942844Z ----------------------------------------------------------------------
2022-05-18T04:17:12.0943249Z Ran 1 test in 0.986s
2022-05-18T04:17:12.0943352Z 
2022-05-18T04:17:12.0943426Z OK (skipped=1)
2022-05-18T04:17:12.0943533Z 
2022-05-18T04:17:12.0943619Z Generating XML reports...
2022-05-18T04:17:12.0976752Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518041711.xml
2022-05-18T04:17:12.8578406Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:17:12.8588086Z 
2022-05-18T04:17:12.8588202Z Running tests...
2022-05-18T04:17:12.8588999Z ----------------------------------------------------------------------
2022-05-18T04:17:12.8595410Z   test_ddp_checkpointing_twice_static_graph_use_reentrant_False (__main__.DistributedDataParallelTest)
2022-05-18T04:17:13.1377528Z Regardless of reentrant or non-reentrant checkpointing impl, ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9165
2022-05-18T04:17:13.1400518Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9166
2022-05-18T04:17:13.7366874Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:17:13.7562054Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:17:13.9426523Z skip: Need at least 2 CUDA devices (1.083s)
2022-05-18T04:17:13.9426816Z 
2022-05-18T04:17:13.9427259Z ----------------------------------------------------------------------
2022-05-18T04:17:13.9427623Z Ran 1 test in 1.084s
2022-05-18T04:17:13.9427810Z 
2022-05-18T04:17:13.9427919Z OK (skipped=1)
2022-05-18T04:17:13.9428086Z 
2022-05-18T04:17:13.9428217Z Generating XML reports...
2022-05-18T04:17:13.9462667Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518041712.xml
2022-05-18T04:17:14.6997579Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:17:14.7006829Z 
2022-05-18T04:17:14.7006930Z Running tests...
2022-05-18T04:17:14.7007500Z ----------------------------------------------------------------------
2022-05-18T04:17:14.7014310Z   test_ddp_checkpointing_twice_static_graph_use_reentrant_True (__main__.DistributedDataParallelTest)
2022-05-18T04:17:14.9797600Z Regardless of reentrant or non-reentrant checkpointing impl, ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9198
2022-05-18T04:17:14.9818735Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9199
2022-05-18T04:17:15.5500096Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:17:15.5537187Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:17:15.6842121Z skip: Need at least 2 CUDA devices (0.983s)
2022-05-18T04:17:15.6842288Z 
2022-05-18T04:17:15.6842605Z ----------------------------------------------------------------------
2022-05-18T04:17:15.6842995Z Ran 1 test in 0.983s
2022-05-18T04:17:15.6843098Z 
2022-05-18T04:17:15.6843173Z OK (skipped=1)
2022-05-18T04:17:15.6843282Z 
2022-05-18T04:17:15.6843370Z Generating XML reports...
2022-05-18T04:17:15.6877201Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518041714.xml
2022-05-18T04:17:16.4455274Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:17:16.4465357Z 
2022-05-18T04:17:16.4465482Z Running tests...
2022-05-18T04:17:16.4465886Z ----------------------------------------------------------------------
2022-05-18T04:17:16.4475610Z   test_ddp_checkpointing_twice_use_reentrant_False (__main__.DistributedDataParallelTest)
2022-05-18T04:17:16.7268010Z Checkpoitning twice fails for non-static graph with reentrant checkpoint ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9231
2022-05-18T04:17:16.7289646Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9232
2022-05-18T04:17:17.3022754Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:17:17.3377429Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:17:17.5314734Z skip: Need at least 2 CUDA devices (1.085s)
2022-05-18T04:17:17.5315002Z 
2022-05-18T04:17:17.5315451Z ----------------------------------------------------------------------
2022-05-18T04:17:17.5315850Z Ran 1 test in 1.085s
2022-05-18T04:17:17.5316020Z 
2022-05-18T04:17:17.5316129Z OK (skipped=1)
2022-05-18T04:17:17.5316291Z 
2022-05-18T04:17:17.5316425Z Generating XML reports...
2022-05-18T04:17:17.5350878Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518041716.xml
2022-05-18T04:17:18.2894041Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:17:18.2903990Z 
2022-05-18T04:17:18.2904268Z Running tests...
2022-05-18T04:17:18.2904884Z ----------------------------------------------------------------------
2022-05-18T04:17:18.2914658Z   test_ddp_checkpointing_twice_use_reentrant_True (__main__.DistributedDataParallelTest)
2022-05-18T04:17:18.5710850Z Checkpoitning twice fails for non-static graph with reentrant checkpoint ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9264
2022-05-18T04:17:18.5732121Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9265
2022-05-18T04:17:19.1493758Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:17:19.1496271Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:17:19.2756532Z skip: Need at least 2 CUDA devices (0.985s)
2022-05-18T04:17:19.2756758Z 
2022-05-18T04:17:19.2757109Z ----------------------------------------------------------------------
2022-05-18T04:17:19.2757631Z Ran 1 test in 0.985s
2022-05-18T04:17:19.2757748Z 
2022-05-18T04:17:19.2757808Z OK (skipped=1)
2022-05-18T04:17:19.2757922Z 
2022-05-18T04:17:19.2758061Z Generating XML reports...
2022-05-18T04:17:19.2791158Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518041718.xml
2022-05-18T04:17:20.0381616Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:17:20.0391775Z 
2022-05-18T04:17:20.0391852Z Running tests...
2022-05-18T04:17:20.0392791Z ----------------------------------------------------------------------
2022-05-18T04:17:20.0398889Z   test_ddp_checkpointing_twice_weight_sharing (__main__.DistributedDataParallelTest)
2022-05-18T04:17:20.3187904Z Checkpointing should work with static graph in the case of checkpointing ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9297
2022-05-18T04:17:20.3210619Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9298
2022-05-18T04:17:20.9033583Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:17:20.9336876Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:17:21.1235659Z skip: Need at least 2 CUDA devices (1.084s)
2022-05-18T04:17:21.1236182Z 
2022-05-18T04:17:21.1236849Z ----------------------------------------------------------------------
2022-05-18T04:17:21.1237338Z Ran 1 test in 1.084s
2022-05-18T04:17:21.1237465Z 
2022-05-18T04:17:21.1237546Z OK (skipped=1)
2022-05-18T04:17:21.1237655Z 
2022-05-18T04:17:21.1237744Z Generating XML reports...
2022-05-18T04:17:21.1270170Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518041720.xml
2022-05-18T04:17:21.8850469Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:17:21.8859748Z 
2022-05-18T04:17:21.8859886Z Running tests...
2022-05-18T04:17:21.8860485Z ----------------------------------------------------------------------
2022-05-18T04:17:21.8869689Z   test_ddp_checkpointing_unused_params_use_reentrant_False (__main__.DistributedDataParallelTest)
2022-05-18T04:17:22.1705912Z With reentrant autograd checkpointing impl, DDP will fail when there are ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9330
2022-05-18T04:17:22.1727769Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9331
2022-05-18T04:17:22.7571123Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:17:22.7624238Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:17:22.8750668Z skip: Need at least 2 CUDA devices (0.989s)
2022-05-18T04:17:22.8750928Z 
2022-05-18T04:17:22.8751382Z ----------------------------------------------------------------------
2022-05-18T04:17:22.8751785Z Ran 1 test in 0.989s
2022-05-18T04:17:22.8751966Z 
2022-05-18T04:17:22.8752081Z OK (skipped=1)
2022-05-18T04:17:22.8752249Z 
2022-05-18T04:17:22.8752391Z Generating XML reports...
2022-05-18T04:17:22.8786470Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518041721.xml
2022-05-18T04:17:23.6371886Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:17:23.6381062Z 
2022-05-18T04:17:23.6381198Z Running tests...
2022-05-18T04:17:23.6381636Z ----------------------------------------------------------------------
2022-05-18T04:17:23.6391225Z   test_ddp_checkpointing_unused_params_use_reentrant_True (__main__.DistributedDataParallelTest)
2022-05-18T04:17:23.9176298Z With reentrant autograd checkpointing impl, DDP will fail when there are ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9363
2022-05-18T04:17:23.9198266Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9364
2022-05-18T04:17:24.5071911Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:17:24.5173968Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:17:24.7223347Z skip: Need at least 2 CUDA devices (1.084s)
2022-05-18T04:17:24.7223616Z 
2022-05-18T04:17:24.7223952Z ----------------------------------------------------------------------
2022-05-18T04:17:24.7224202Z Ran 1 test in 1.084s
2022-05-18T04:17:24.7224315Z 
2022-05-18T04:17:24.7224375Z OK (skipped=1)
2022-05-18T04:17:24.7224484Z 
2022-05-18T04:17:24.7224572Z Generating XML reports...
2022-05-18T04:17:24.7258071Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518041723.xml
2022-05-18T04:17:25.4821354Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:17:25.4831242Z 
2022-05-18T04:17:25.4831348Z Running tests...
2022-05-18T04:17:25.4831774Z ----------------------------------------------------------------------
2022-05-18T04:17:25.4842466Z   test_ddp_checkpointing_weight_sharing_use_reentrant_False (__main__.DistributedDataParallelTest)
2022-05-18T04:17:25.7628036Z Test that checkpointing with weight sharing works. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9396
2022-05-18T04:17:25.7650794Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9397
2022-05-18T04:17:26.3335544Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:17:26.3370122Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:17:26.4674552Z skip: Need at least 2 CUDA devices (0.984s)
2022-05-18T04:17:26.4674748Z 
2022-05-18T04:17:26.4675119Z ----------------------------------------------------------------------
2022-05-18T04:17:26.4675373Z Ran 1 test in 0.984s
2022-05-18T04:17:26.4675489Z 
2022-05-18T04:17:26.4675571Z OK (skipped=1)
2022-05-18T04:17:26.4675682Z 
2022-05-18T04:17:26.4675770Z Generating XML reports...
2022-05-18T04:17:26.4708599Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518041725.xml
2022-05-18T04:17:27.2279785Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:17:27.2289402Z 
2022-05-18T04:17:27.2289568Z Running tests...
2022-05-18T04:17:27.2289934Z ----------------------------------------------------------------------
2022-05-18T04:17:27.2300324Z   test_ddp_checkpointing_weight_sharing_use_reentrant_True (__main__.DistributedDataParallelTest)
2022-05-18T04:17:27.5117267Z Test that checkpointing with weight sharing works. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9429
2022-05-18T04:17:27.5139566Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9430
2022-05-18T04:17:28.0845880Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:17:28.1144992Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:17:28.3164417Z skip: Need at least 2 CUDA devices (1.087s)
2022-05-18T04:17:28.3164628Z 
2022-05-18T04:17:28.3165092Z ----------------------------------------------------------------------
2022-05-18T04:17:28.3165550Z Ran 1 test in 1.087s
2022-05-18T04:17:28.3165738Z 
2022-05-18T04:17:28.3165800Z OK (skipped=1)
2022-05-18T04:17:28.3165911Z 
2022-05-18T04:17:28.3165997Z Generating XML reports...
2022-05-18T04:17:28.3199418Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518041727.xml
2022-05-18T04:17:29.0830372Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:17:29.0840765Z 
2022-05-18T04:17:29.0840901Z Running tests...
2022-05-18T04:17:29.0842070Z ----------------------------------------------------------------------
2022-05-18T04:17:29.0845220Z   test_ddp_comm_hook_allreduce_hook_nccl (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-05-18T04:17:29.0845634Z 
2022-05-18T04:17:29.0845891Z ----------------------------------------------------------------------
2022-05-18T04:17:29.0846148Z Ran 1 test in 0.000s
2022-05-18T04:17:29.0846262Z 
2022-05-18T04:17:29.0846336Z OK (skipped=1)
2022-05-18T04:17:29.0846446Z 
2022-05-18T04:17:29.0846518Z Generating XML reports...
2022-05-18T04:17:29.0871413Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518041729.xml
2022-05-18T04:17:29.7617627Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:17:29.7628485Z 
2022-05-18T04:17:29.7628929Z Running tests...
2022-05-18T04:17:29.7629354Z ----------------------------------------------------------------------
2022-05-18T04:17:29.7633442Z   test_ddp_comm_hook_allreduce_hook_nccl_grad_is_view (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-05-18T04:17:29.7633946Z 
2022-05-18T04:17:29.7634290Z ----------------------------------------------------------------------
2022-05-18T04:17:29.7634541Z Ran 1 test in 0.000s
2022-05-18T04:17:29.7634657Z 
2022-05-18T04:17:29.7634731Z OK (skipped=1)
2022-05-18T04:17:29.7634829Z 
2022-05-18T04:17:29.7634916Z Generating XML reports...
2022-05-18T04:17:29.7670107Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518041729.xml
2022-05-18T04:17:30.4328984Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:17:30.4338774Z 
2022-05-18T04:17:30.4339136Z Running tests...
2022-05-18T04:17:30.4339805Z ----------------------------------------------------------------------
2022-05-18T04:17:30.4343041Z   test_ddp_comm_hook_allreduce_hook_nccl_static_graph (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-05-18T04:17:30.4343493Z 
2022-05-18T04:17:30.4343903Z ----------------------------------------------------------------------
2022-05-18T04:17:30.4344351Z Ran 1 test in 0.000s
2022-05-18T04:17:30.4344492Z 
2022-05-18T04:17:30.4344553Z OK (skipped=1)
2022-05-18T04:17:30.4344662Z 
2022-05-18T04:17:30.4344747Z Generating XML reports...
2022-05-18T04:17:30.4368615Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518041730.xml
2022-05-18T04:17:31.1068374Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:17:31.1077659Z 
2022-05-18T04:17:31.1078077Z Running tests...
2022-05-18T04:17:31.1078486Z ----------------------------------------------------------------------
2022-05-18T04:17:31.1089211Z   test_ddp_comm_hook_allreduce_with_then_hook_nccl (__main__.DistributedDataParallelTest)
2022-05-18T04:17:31.1089962Z This unit test verifies whether a DDP communication hook that calls allreduce and then ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-05-18T04:17:31.1090382Z 
2022-05-18T04:17:31.1090803Z ----------------------------------------------------------------------
2022-05-18T04:17:31.1091165Z Ran 1 test in 0.001s
2022-05-18T04:17:31.1091279Z 
2022-05-18T04:17:31.1091353Z OK (skipped=1)
2022-05-18T04:17:31.1091461Z 
2022-05-18T04:17:31.1091545Z Generating XML reports...
2022-05-18T04:17:31.1115802Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518041731.xml
2022-05-18T04:17:31.7780879Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:17:31.7790992Z 
2022-05-18T04:17:31.7791464Z Running tests...
2022-05-18T04:17:31.7792019Z ----------------------------------------------------------------------
2022-05-18T04:17:31.7799026Z   test_ddp_comm_hook_future_passing_gpu_nccl (__main__.DistributedDataParallelTest)
2022-05-18T04:17:31.7799807Z This unit test verifies whether the Future object is passed properly using nccl backend. ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-05-18T04:17:31.7800374Z 
2022-05-18T04:17:31.7800718Z ----------------------------------------------------------------------
2022-05-18T04:17:31.7800960Z Ran 1 test in 0.001s
2022-05-18T04:17:31.7801075Z 
2022-05-18T04:17:31.7801152Z OK (skipped=1)
2022-05-18T04:17:31.7801312Z 
2022-05-18T04:17:31.7801397Z Generating XML reports...
2022-05-18T04:17:31.7830990Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518041731.xml
2022-05-18T04:17:32.4547063Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:17:32.4557080Z 
2022-05-18T04:17:32.4557298Z Running tests...
2022-05-18T04:17:32.4557916Z ----------------------------------------------------------------------
2022-05-18T04:17:32.4569782Z   test_ddp_multi_device_module_config (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-05-18T04:17:32.4570071Z 
2022-05-18T04:17:32.4570414Z ----------------------------------------------------------------------
2022-05-18T04:17:32.4570667Z Ran 1 test in 0.001s
2022-05-18T04:17:32.4570782Z 
2022-05-18T04:17:32.4570856Z OK (skipped=1)
2022-05-18T04:17:32.4570966Z 
2022-05-18T04:17:32.4571038Z Generating XML reports...
2022-05-18T04:17:32.4595114Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518041732.xml
2022-05-18T04:17:33.1272630Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:17:33.1282156Z 
2022-05-18T04:17:33.1282609Z Running tests...
2022-05-18T04:17:33.1292235Z ----------------------------------------------------------------------
2022-05-18T04:17:33.1296301Z   test_ddp_weight_sharing (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-05-18T04:17:33.1297008Z 
2022-05-18T04:17:33.1297525Z ----------------------------------------------------------------------
2022-05-18T04:17:33.1297877Z Ran 1 test in 0.001s
2022-05-18T04:17:33.1298003Z 
2022-05-18T04:17:33.1298080Z OK (skipped=1)
2022-05-18T04:17:33.1298188Z 
2022-05-18T04:17:33.1298281Z Generating XML reports...
2022-05-18T04:17:33.1321747Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518041733.xml
2022-05-18T04:17:33.8011019Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:17:33.8020884Z 
2022-05-18T04:17:33.8021114Z Running tests...
2022-05-18T04:17:33.8021721Z ----------------------------------------------------------------------
2022-05-18T04:17:33.8027748Z   test_ddp_with_lazy_parameters (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-05-18T04:17:33.8028217Z 
2022-05-18T04:17:33.8028667Z ----------------------------------------------------------------------
2022-05-18T04:17:33.8028935Z Ran 1 test in 0.001s
2022-05-18T04:17:33.8029050Z 
2022-05-18T04:17:33.8029110Z OK (skipped=1)
2022-05-18T04:17:33.8029219Z 
2022-05-18T04:17:33.8029304Z Generating XML reports...
2022-05-18T04:17:33.8060024Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518041733.xml
2022-05-18T04:17:34.4756452Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:17:34.4766116Z 
2022-05-18T04:17:34.4766202Z Running tests...
2022-05-18T04:17:34.4766667Z ----------------------------------------------------------------------
2022-05-18T04:17:34.4770474Z   test_default_ddp_comm_hooks_nccl (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-05-18T04:17:34.4770773Z 
2022-05-18T04:17:34.4771325Z ----------------------------------------------------------------------
2022-05-18T04:17:34.4771748Z Ran 1 test in 0.000s
2022-05-18T04:17:34.4771928Z 
2022-05-18T04:17:34.4772041Z OK (skipped=1)
2022-05-18T04:17:34.4772206Z 
2022-05-18T04:17:34.4773819Z Generating XML reports...
2022-05-18T04:17:34.4796803Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518041734.xml
2022-05-18T04:17:35.1459938Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:17:35.1470174Z 
2022-05-18T04:17:35.1470260Z Running tests...
2022-05-18T04:17:35.1470861Z ----------------------------------------------------------------------
2022-05-18T04:17:35.1474639Z   test_default_ddp_comm_hooks_nccl_is_view (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-05-18T04:17:35.1475068Z 
2022-05-18T04:17:35.1475397Z ----------------------------------------------------------------------
2022-05-18T04:17:35.1475676Z Ran 1 test in 0.000s
2022-05-18T04:17:35.1475792Z 
2022-05-18T04:17:35.1475865Z OK (skipped=1)
2022-05-18T04:17:35.1475973Z 
2022-05-18T04:17:35.1476064Z Generating XML reports...
2022-05-18T04:17:35.1499759Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518041735.xml
2022-05-18T04:17:35.8160970Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:17:35.8170360Z 
2022-05-18T04:17:35.8170494Z Running tests...
2022-05-18T04:17:35.8171086Z ----------------------------------------------------------------------
2022-05-18T04:17:35.8191950Z   test_failure_recovery (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.002s)
2022-05-18T04:17:35.8192581Z 
2022-05-18T04:17:35.8193034Z ----------------------------------------------------------------------
2022-05-18T04:17:35.8193294Z Ran 1 test in 0.002s
2022-05-18T04:17:35.8193409Z 
2022-05-18T04:17:35.8193479Z OK (skipped=1)
2022-05-18T04:17:35.8193588Z 
2022-05-18T04:17:35.8193677Z Generating XML reports...
2022-05-18T04:17:35.8222592Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518041735.xml
2022-05-18T04:17:36.4896153Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:17:36.4906022Z 
2022-05-18T04:17:36.4906124Z Running tests...
2022-05-18T04:17:36.4906788Z ----------------------------------------------------------------------
2022-05-18T04:17:36.4911216Z   test_find_unused_parameters_kwarg_debug_detail (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-05-18T04:17:36.4911633Z 
2022-05-18T04:17:36.4912073Z ----------------------------------------------------------------------
2022-05-18T04:17:36.4912444Z Ran 1 test in 0.000s
2022-05-18T04:17:36.4912560Z 
2022-05-18T04:17:36.4912634Z OK (skipped=1)
2022-05-18T04:17:36.4912735Z 
2022-05-18T04:17:36.4912822Z Generating XML reports...
2022-05-18T04:17:36.4936731Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518041736.xml
2022-05-18T04:17:37.1589326Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:17:37.1598988Z 
2022-05-18T04:17:37.1599087Z Running tests...
2022-05-18T04:17:37.1599656Z ----------------------------------------------------------------------
2022-05-18T04:17:37.1603885Z   test_find_unused_parameters_kwarg_debug_info (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-05-18T04:17:37.1604316Z 
2022-05-18T04:17:37.1604747Z ----------------------------------------------------------------------
2022-05-18T04:17:37.1605379Z Ran 1 test in 0.000s
2022-05-18T04:17:37.1605496Z 
2022-05-18T04:17:37.1605569Z OK (skipped=1)
2022-05-18T04:17:37.1605678Z 
2022-05-18T04:17:37.1605829Z Generating XML reports...
2022-05-18T04:17:37.1630768Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518041737.xml
2022-05-18T04:17:37.8314457Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:17:37.8323703Z 
2022-05-18T04:17:37.8323813Z Running tests...
2022-05-18T04:17:37.8324407Z ----------------------------------------------------------------------
2022-05-18T04:17:37.8329178Z   test_find_unused_parameters_kwarg_debug_off (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-05-18T04:17:37.8329687Z 
2022-05-18T04:17:37.8330080Z ----------------------------------------------------------------------
2022-05-18T04:17:37.8330329Z Ran 1 test in 0.000s
2022-05-18T04:17:37.8330446Z 
2022-05-18T04:17:37.8330519Z OK (skipped=1)
2022-05-18T04:17:37.8330628Z 
2022-05-18T04:17:37.8330714Z Generating XML reports...
2022-05-18T04:17:37.8359894Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518041737.xml
2022-05-18T04:17:38.5025625Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:17:38.5035275Z 
2022-05-18T04:17:38.5035372Z Running tests...
2022-05-18T04:17:38.5036298Z ----------------------------------------------------------------------
2022-05-18T04:17:38.5041569Z   test_find_unused_parameters_kwarg_grad_is_view_debug_detail (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-05-18T04:17:38.5042057Z 
2022-05-18T04:17:38.5042437Z ----------------------------------------------------------------------
2022-05-18T04:17:38.5042911Z Ran 1 test in 0.001s
2022-05-18T04:17:38.5043122Z 
2022-05-18T04:17:38.5043224Z OK (skipped=1)
2022-05-18T04:17:38.5043407Z 
2022-05-18T04:17:38.5043553Z Generating XML reports...
2022-05-18T04:17:38.5067650Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518041738.xml
2022-05-18T04:17:39.1716970Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:17:39.1727338Z 
2022-05-18T04:17:39.1727795Z Running tests...
2022-05-18T04:17:39.1728203Z ----------------------------------------------------------------------
2022-05-18T04:17:39.1732454Z   test_find_unused_parameters_kwarg_grad_is_view_debug_info (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-05-18T04:17:39.1732945Z 
2022-05-18T04:17:39.1733348Z ----------------------------------------------------------------------
2022-05-18T04:17:39.1733737Z Ran 1 test in 0.000s
2022-05-18T04:17:39.1733932Z 
2022-05-18T04:17:39.1734057Z OK (skipped=1)
2022-05-18T04:17:39.1734258Z 
2022-05-18T04:17:39.1734400Z Generating XML reports...
2022-05-18T04:17:39.1758857Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518041739.xml
2022-05-18T04:17:39.8399024Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:17:39.8409257Z 
2022-05-18T04:17:39.8409550Z Running tests...
2022-05-18T04:17:39.8410168Z ----------------------------------------------------------------------
2022-05-18T04:17:39.8414596Z   test_find_unused_parameters_kwarg_grad_is_view_debug_off (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-05-18T04:17:39.8414937Z 
2022-05-18T04:17:39.8415222Z ----------------------------------------------------------------------
2022-05-18T04:17:39.8415488Z Ran 1 test in 0.001s
2022-05-18T04:17:39.8415632Z 
2022-05-18T04:17:39.8415749Z OK (skipped=1)
2022-05-18T04:17:39.8415859Z 
2022-05-18T04:17:39.8416174Z Generating XML reports...
2022-05-18T04:17:39.8447411Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518041739.xml
2022-05-18T04:17:40.5119511Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:17:40.5129381Z 
2022-05-18T04:17:40.5129681Z Running tests...
2022-05-18T04:17:40.5130336Z ----------------------------------------------------------------------
2022-05-18T04:17:40.5133617Z   test_fp16 (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-05-18T04:17:40.5134037Z 
2022-05-18T04:17:40.5134433Z ----------------------------------------------------------------------
2022-05-18T04:17:40.5134840Z Ran 1 test in 0.000s
2022-05-18T04:17:40.5135034Z 
2022-05-18T04:17:40.5135156Z OK (skipped=1)
2022-05-18T04:17:40.5135321Z 
2022-05-18T04:17:40.5135465Z Generating XML reports...
2022-05-18T04:17:40.5159948Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518041740.xml
2022-05-18T04:17:41.1827956Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:17:41.1838258Z 
2022-05-18T04:17:41.1838958Z Running tests...
2022-05-18T04:17:41.1839376Z ----------------------------------------------------------------------
2022-05-18T04:17:41.1843143Z   test_fp16_compress_wrapper_is_view (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-05-18T04:17:41.1843554Z 
2022-05-18T04:17:41.1843986Z ----------------------------------------------------------------------
2022-05-18T04:17:41.1844380Z Ran 1 test in 0.000s
2022-05-18T04:17:41.1844495Z 
2022-05-18T04:17:41.1844569Z OK (skipped=1)
2022-05-18T04:17:41.1844676Z 
2022-05-18T04:17:41.1844760Z Generating XML reports...
2022-05-18T04:17:41.1868109Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518041741.xml
2022-05-18T04:17:41.8522690Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:17:41.8532647Z 
2022-05-18T04:17:41.8533011Z Running tests...
2022-05-18T04:17:41.8533703Z ----------------------------------------------------------------------
2022-05-18T04:17:41.8537048Z   test_fp16_compress_wrapper_nccl (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-05-18T04:17:41.8537519Z 
2022-05-18T04:17:41.8538069Z ----------------------------------------------------------------------
2022-05-18T04:17:41.8538539Z Ran 1 test in 0.000s
2022-05-18T04:17:41.8538731Z 
2022-05-18T04:17:41.8538813Z OK (skipped=1)
2022-05-18T04:17:41.8538926Z 
2022-05-18T04:17:41.8539012Z Generating XML reports...
2022-05-18T04:17:41.8569692Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518041741.xml
2022-05-18T04:17:42.5274293Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:17:42.5283726Z 
2022-05-18T04:17:42.5283875Z Running tests...
2022-05-18T04:17:42.5284477Z ----------------------------------------------------------------------
2022-05-18T04:17:42.5288550Z   test_fp16_grad_is_view (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-05-18T04:17:42.5288915Z 
2022-05-18T04:17:42.5289300Z ----------------------------------------------------------------------
2022-05-18T04:17:42.5289784Z Ran 1 test in 0.000s
2022-05-18T04:17:42.5289923Z 
2022-05-18T04:17:42.5289983Z OK (skipped=1)
2022-05-18T04:17:42.5290092Z 
2022-05-18T04:17:42.5290177Z Generating XML reports...
2022-05-18T04:17:42.5314033Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518041742.xml
2022-05-18T04:17:43.1968297Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:17:43.1978402Z 
2022-05-18T04:17:43.1978710Z Running tests...
2022-05-18T04:17:43.1979090Z ----------------------------------------------------------------------
2022-05-18T04:17:43.1984597Z   test_grad_layout_1devicemodule_1replicaperprocess (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-05-18T04:17:43.1985088Z 
2022-05-18T04:17:43.1985456Z ----------------------------------------------------------------------
2022-05-18T04:17:43.1985910Z Ran 1 test in 0.001s
2022-05-18T04:17:43.1986081Z 
2022-05-18T04:17:43.1986158Z OK (skipped=1)
2022-05-18T04:17:43.1986264Z 
2022-05-18T04:17:43.1986348Z Generating XML reports...
2022-05-18T04:17:43.2010081Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518041743.xml
2022-05-18T04:17:43.8713087Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:17:43.8723017Z 
2022-05-18T04:17:43.8723206Z Running tests...
2022-05-18T04:17:43.8723588Z ----------------------------------------------------------------------
2022-05-18T04:17:43.8730681Z   test_grad_layout_2devicemodule (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-05-18T04:17:43.8731189Z 
2022-05-18T04:17:43.8731534Z ----------------------------------------------------------------------
2022-05-18T04:17:43.8731785Z Ran 1 test in 0.001s
2022-05-18T04:17:43.8731901Z 
2022-05-18T04:17:43.8731960Z OK (skipped=1)
2022-05-18T04:17:43.8732068Z 
2022-05-18T04:17:43.8732154Z Generating XML reports...
2022-05-18T04:17:43.8761327Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518041743.xml
2022-05-18T04:17:44.5442699Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:17:44.5452403Z 
2022-05-18T04:17:44.5452503Z Running tests...
2022-05-18T04:17:44.5452989Z ----------------------------------------------------------------------
2022-05-18T04:17:44.8261862Z   test_invalid_powerSGD_state (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9692
2022-05-18T04:17:44.8283443Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9693
2022-05-18T04:17:45.3934036Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:17:45.3940134Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-05-18T04:17:45.3941169Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:17:45.3941843Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-05-18T04:17:45.3942670Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-05-18T04:17:45.3943737Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-05-18T04:17:45.3947132Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-05-18T04:17:45.3948174Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-05-18T04:17:45.3948987Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-05-18T04:17:45.3949797Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-05-18T04:17:45.3950599Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-05-18T04:17:45.3951398Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-05-18T04:17:45.3952205Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-05-18T04:17:45.3953008Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-05-18T04:17:45.5306423Z ok (0.985s)
2022-05-18T04:17:45.5306660Z 
2022-05-18T04:17:45.5307120Z ----------------------------------------------------------------------
2022-05-18T04:17:45.5307521Z Ran 1 test in 0.985s
2022-05-18T04:17:45.5307704Z 
2022-05-18T04:17:45.5307813Z OK
2022-05-18T04:17:45.5307952Z 
2022-05-18T04:17:45.5308108Z Generating XML reports...
2022-05-18T04:17:45.5342194Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518041744.xml
2022-05-18T04:17:46.2937582Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:17:46.2947619Z 
2022-05-18T04:17:46.2947861Z Running tests...
2022-05-18T04:17:46.2948290Z ----------------------------------------------------------------------
2022-05-18T04:17:46.2951939Z   test_multiple_outputs_multiple_backward (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-05-18T04:17:46.2952395Z 
2022-05-18T04:17:46.2952811Z ----------------------------------------------------------------------
2022-05-18T04:17:46.2953508Z Ran 1 test in 0.000s
2022-05-18T04:17:46.2953703Z 
2022-05-18T04:17:46.2953827Z OK (skipped=1)
2022-05-18T04:17:46.2953999Z 
2022-05-18T04:17:46.2954143Z Generating XML reports...
2022-05-18T04:17:46.2978518Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518041746.xml
2022-05-18T04:17:46.9652335Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:17:46.9662352Z 
2022-05-18T04:17:46.9662824Z Running tests...
2022-05-18T04:17:46.9663406Z ----------------------------------------------------------------------
2022-05-18T04:17:46.9667284Z   test_multiple_outputs_multiple_backward_grad_is_view (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-05-18T04:17:46.9667765Z 
2022-05-18T04:17:46.9668168Z ----------------------------------------------------------------------
2022-05-18T04:17:46.9668574Z Ran 1 test in 0.000s
2022-05-18T04:17:46.9668786Z 
2022-05-18T04:17:46.9668896Z OK (skipped=1)
2022-05-18T04:17:46.9669074Z 
2022-05-18T04:17:46.9669223Z Generating XML reports...
2022-05-18T04:17:46.9705400Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518041746.xml
2022-05-18T04:17:47.6373192Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:17:47.6383752Z 
2022-05-18T04:17:47.6384180Z Running tests...
2022-05-18T04:17:47.6384577Z ----------------------------------------------------------------------
2022-05-18T04:17:47.6389987Z   test_nccl_backend_1gpu_module_device_ids_integer_list (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-05-18T04:17:47.6390488Z 
2022-05-18T04:17:47.6390951Z ----------------------------------------------------------------------
2022-05-18T04:17:47.6391410Z Ran 1 test in 0.001s
2022-05-18T04:17:47.6391608Z 
2022-05-18T04:17:47.6391768Z OK (skipped=1)
2022-05-18T04:17:47.6391967Z 
2022-05-18T04:17:47.6392103Z Generating XML reports...
2022-05-18T04:17:47.6415362Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518041747.xml
2022-05-18T04:17:48.3062307Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:17:48.3071920Z 
2022-05-18T04:17:48.3072061Z Running tests...
2022-05-18T04:17:48.3072538Z ----------------------------------------------------------------------
2022-05-18T04:17:48.3077608Z   test_nccl_backend_1gpu_module_device_ids_torch_device_list (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-05-18T04:17:48.3078047Z 
2022-05-18T04:17:48.3078392Z ----------------------------------------------------------------------
2022-05-18T04:17:48.3078828Z Ran 1 test in 0.001s
2022-05-18T04:17:48.3079032Z 
2022-05-18T04:17:48.3079145Z OK (skipped=1)
2022-05-18T04:17:48.3079345Z 
2022-05-18T04:17:48.3079502Z Generating XML reports...
2022-05-18T04:17:48.3104314Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518041748.xml
2022-05-18T04:17:48.9766435Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:17:48.9776034Z 
2022-05-18T04:17:48.9776124Z Running tests...
2022-05-18T04:17:48.9776779Z ----------------------------------------------------------------------
2022-05-18T04:17:48.9782655Z   test_nccl_backend_2gpu_module (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-05-18T04:17:48.9783125Z 
2022-05-18T04:17:48.9783453Z ----------------------------------------------------------------------
2022-05-18T04:17:48.9783708Z Ran 1 test in 0.001s
2022-05-18T04:17:48.9783847Z 
2022-05-18T04:17:48.9783964Z OK (skipped=1)
2022-05-18T04:17:48.9784061Z 
2022-05-18T04:17:48.9784147Z Generating XML reports...
2022-05-18T04:17:48.9815511Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518041748.xml
2022-05-18T04:17:49.6518074Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:17:49.6527571Z 
2022-05-18T04:17:49.6527692Z Running tests...
2022-05-18T04:17:49.6528266Z ----------------------------------------------------------------------
2022-05-18T04:17:49.6533752Z   test_nccl_backend_4gpu_module (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-05-18T04:17:49.6534200Z 
2022-05-18T04:17:49.6534550Z ----------------------------------------------------------------------
2022-05-18T04:17:49.6534776Z Ran 1 test in 0.001s
2022-05-18T04:17:49.6534890Z 
2022-05-18T04:17:49.6534963Z OK (skipped=1)
2022-05-18T04:17:49.6535072Z 
2022-05-18T04:17:49.6535157Z Generating XML reports...
2022-05-18T04:17:49.6559128Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518041749.xml
2022-05-18T04:17:50.3225237Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:17:50.3234243Z 
2022-05-18T04:17:50.3234395Z Running tests...
2022-05-18T04:17:50.3234848Z ----------------------------------------------------------------------
2022-05-18T04:17:50.3241019Z   test_nccl_backend_multi_device_ids_not_allowed (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-05-18T04:17:50.3241623Z 
2022-05-18T04:17:50.3242074Z ----------------------------------------------------------------------
2022-05-18T04:17:50.3242328Z Ran 1 test in 0.001s
2022-05-18T04:17:50.3242520Z 
2022-05-18T04:17:50.3242595Z OK (skipped=1)
2022-05-18T04:17:50.3242706Z 
2022-05-18T04:17:50.3242780Z Generating XML reports...
2022-05-18T04:17:50.3266373Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518041750.xml
2022-05-18T04:17:50.9922603Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:17:50.9932437Z 
2022-05-18T04:17:50.9932579Z Running tests...
2022-05-18T04:17:50.9932966Z ----------------------------------------------------------------------
2022-05-18T04:17:50.9938568Z   test_nccl_backend_multi_device_module_device_ids_None (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-05-18T04:17:50.9939607Z 
2022-05-18T04:17:50.9939830Z ----------------------------------------------------------------------
2022-05-18T04:17:50.9940076Z Ran 1 test in 0.001s
2022-05-18T04:17:50.9940178Z 
2022-05-18T04:17:50.9940251Z OK (skipped=1)
2022-05-18T04:17:50.9940361Z 
2022-05-18T04:17:50.9940448Z Generating XML reports...
2022-05-18T04:17:50.9971665Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518041750.xml
2022-05-18T04:17:51.6617254Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:17:51.6628113Z 
2022-05-18T04:17:51.6628330Z Running tests...
2022-05-18T04:17:51.6628692Z ----------------------------------------------------------------------
2022-05-18T04:17:51.6632616Z   test_nccl_backend_single_device_module_device_ids_None (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-05-18T04:17:51.6632942Z 
2022-05-18T04:17:51.6633312Z ----------------------------------------------------------------------
2022-05-18T04:17:51.6633767Z Ran 1 test in 0.000s
2022-05-18T04:17:51.6633975Z 
2022-05-18T04:17:51.6634089Z OK (skipped=1)
2022-05-18T04:17:51.6634202Z 
2022-05-18T04:17:51.6634287Z Generating XML reports...
2022-05-18T04:17:51.6659052Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518041751.xml
2022-05-18T04:17:52.3364363Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:17:52.3374490Z 
2022-05-18T04:17:52.3374648Z Running tests...
2022-05-18T04:17:52.3375442Z ----------------------------------------------------------------------
2022-05-18T04:17:52.3379505Z   test_nccl_backend_single_device_module_empty_device_ids (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-05-18T04:17:52.3380002Z 
2022-05-18T04:17:52.3380398Z ----------------------------------------------------------------------
2022-05-18T04:17:52.3380813Z Ran 1 test in 0.000s
2022-05-18T04:17:52.3381002Z 
2022-05-18T04:17:52.3381117Z OK (skipped=1)
2022-05-18T04:17:52.3381294Z 
2022-05-18T04:17:52.3381436Z Generating XML reports...
2022-05-18T04:17:52.3406103Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518041752.xml
2022-05-18T04:17:53.0091558Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:17:53.0101165Z 
2022-05-18T04:17:53.0101308Z Running tests...
2022-05-18T04:17:53.0101748Z ----------------------------------------------------------------------
2022-05-18T04:17:53.0115640Z   test_nccl_propagate_error_reason (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-05-18T04:17:53.0116126Z 
2022-05-18T04:17:53.0116419Z ----------------------------------------------------------------------
2022-05-18T04:17:53.0116654Z Ran 1 test in 0.001s
2022-05-18T04:17:53.0116767Z 
2022-05-18T04:17:53.0116842Z OK (skipped=1)
2022-05-18T04:17:53.0116950Z 
2022-05-18T04:17:53.0117035Z Generating XML reports...
2022-05-18T04:17:53.0140635Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518041753.xml
2022-05-18T04:17:53.6823798Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:17:53.6833732Z 
2022-05-18T04:17:53.6833873Z Running tests...
2022-05-18T04:17:53.6834292Z ----------------------------------------------------------------------
2022-05-18T04:17:53.6850180Z   test_no_grad (__main__.DistributedDataParallelTest)
2022-05-18T04:17:53.6851402Z Note: this test can be sped up by only running it on a CPU module ... skip: c10d was not compiled with the NCCL backend (0.002s)
2022-05-18T04:17:53.6851783Z 
2022-05-18T04:17:53.6852244Z ----------------------------------------------------------------------
2022-05-18T04:17:53.6852615Z Ran 1 test in 0.002s
2022-05-18T04:17:53.6852716Z 
2022-05-18T04:17:53.6852792Z OK (skipped=1)
2022-05-18T04:17:53.6852900Z 
2022-05-18T04:17:53.6852985Z Generating XML reports...
2022-05-18T04:17:53.6882454Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518041753.xml
2022-05-18T04:17:54.3595840Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:17:54.3605474Z 
2022-05-18T04:17:54.3605571Z Running tests...
2022-05-18T04:17:54.3606319Z ----------------------------------------------------------------------
2022-05-18T04:17:54.3616175Z   test_param_layout_mismatch_error (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-05-18T04:17:54.3616567Z 
2022-05-18T04:17:54.3616865Z ----------------------------------------------------------------------
2022-05-18T04:17:54.3617099Z Ran 1 test in 0.001s
2022-05-18T04:17:54.3617215Z 
2022-05-18T04:17:54.3617289Z OK (skipped=1)
2022-05-18T04:17:54.3617397Z 
2022-05-18T04:17:54.3617484Z Generating XML reports...
2022-05-18T04:17:54.3641358Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518041754.xml
2022-05-18T04:17:55.0332374Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:17:55.0343125Z 
2022-05-18T04:17:55.0343398Z Running tests...
2022-05-18T04:17:55.0344331Z ----------------------------------------------------------------------
2022-05-18T04:17:55.0350011Z   test_pass_default_pg (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-05-18T04:17:55.0350606Z 
2022-05-18T04:17:55.0351010Z ----------------------------------------------------------------------
2022-05-18T04:17:55.0351420Z Ran 1 test in 0.001s
2022-05-18T04:17:55.0351607Z 
2022-05-18T04:17:55.0351735Z OK (skipped=1)
2022-05-18T04:17:55.0351900Z 
2022-05-18T04:17:55.0352045Z Generating XML reports...
2022-05-18T04:17:55.0376147Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518041755.xml
2022-05-18T04:17:55.7074775Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:17:55.7085529Z 
2022-05-18T04:17:55.7085860Z Running tests...
2022-05-18T04:17:55.7086500Z ----------------------------------------------------------------------
2022-05-18T04:17:55.7090337Z   test_powerSGD_ddp_comm_hook_nccl (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-05-18T04:17:55.7090821Z 
2022-05-18T04:17:55.7091259Z ----------------------------------------------------------------------
2022-05-18T04:17:55.7091686Z Ran 1 test in 0.000s
2022-05-18T04:17:55.7091893Z 
2022-05-18T04:17:55.7092022Z OK (skipped=1)
2022-05-18T04:17:55.7092220Z 
2022-05-18T04:17:55.7092357Z Generating XML reports...
2022-05-18T04:17:55.7123780Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518041755.xml
2022-05-18T04:17:56.3810364Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:17:56.3819873Z 
2022-05-18T04:17:56.3820177Z Running tests...
2022-05-18T04:17:56.3820776Z ----------------------------------------------------------------------
2022-05-18T04:17:56.3824232Z   test_powerSGD_ddp_comm_hook_nccl_grad_is_view (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-05-18T04:17:56.3824737Z 
2022-05-18T04:17:56.3825196Z ----------------------------------------------------------------------
2022-05-18T04:17:56.3825521Z Ran 1 test in 0.000s
2022-05-18T04:17:56.3825636Z 
2022-05-18T04:17:56.3825700Z OK (skipped=1)
2022-05-18T04:17:56.3825813Z 
2022-05-18T04:17:56.3825900Z Generating XML reports...
2022-05-18T04:17:56.3849086Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518041756.xml
2022-05-18T04:17:57.0511197Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:17:57.0519731Z 
2022-05-18T04:17:57.0519828Z Running tests...
2022-05-18T04:17:57.0520276Z ----------------------------------------------------------------------
2022-05-18T04:17:57.3329183Z   test_sync_batch_norm_empty_input (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9885
2022-05-18T04:17:57.3350934Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9886
2022-05-18T04:17:57.9018736Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:17:57.9398003Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:17:58.1375195Z skip: Need at least 2 CUDA devices (1.085s)
2022-05-18T04:17:58.1375772Z 
2022-05-18T04:17:58.1376396Z ----------------------------------------------------------------------
2022-05-18T04:17:58.1376874Z Ran 1 test in 1.086s
2022-05-18T04:17:58.1377084Z 
2022-05-18T04:17:58.1377195Z OK (skipped=1)
2022-05-18T04:17:58.1377318Z 
2022-05-18T04:17:58.1377406Z Generating XML reports...
2022-05-18T04:17:58.1412620Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518041757.xml
2022-05-18T04:17:58.9092676Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:17:58.9101672Z 
2022-05-18T04:17:58.9101776Z Running tests...
2022-05-18T04:17:58.9102333Z ----------------------------------------------------------------------
2022-05-18T04:17:59.1898274Z   test_sync_batch_norm_only_empty_input (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9918
2022-05-18T04:17:59.1920424Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9919
2022-05-18T04:17:59.7588869Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:17:59.7666956Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:17:59.8943797Z skip: Need at least 2 CUDA devices (0.984s)
2022-05-18T04:17:59.8944214Z 
2022-05-18T04:17:59.8944671Z ----------------------------------------------------------------------
2022-05-18T04:17:59.8944942Z Ran 1 test in 0.984s
2022-05-18T04:17:59.8945081Z 
2022-05-18T04:17:59.8945157Z OK (skipped=1)
2022-05-18T04:17:59.8945273Z 
2022-05-18T04:17:59.8945393Z Generating XML reports...
2022-05-18T04:17:59.8978919Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518041758.xml
2022-05-18T04:18:00.6619043Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:18:00.6628695Z 
2022-05-18T04:18:00.6628894Z Running tests...
2022-05-18T04:18:00.6629357Z ----------------------------------------------------------------------
2022-05-18T04:18:00.6633877Z   test_invalid_nccl_blocking_wait_env (__main__.NcclErrorHandlingTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-05-18T04:18:00.6634254Z 
2022-05-18T04:18:00.6634962Z ----------------------------------------------------------------------
2022-05-18T04:18:00.6635451Z Ran 1 test in 0.001s
2022-05-18T04:18:00.6635600Z 
2022-05-18T04:18:00.6635665Z OK (skipped=1)
2022-05-18T04:18:00.6635799Z 
2022-05-18T04:18:00.6635885Z Generating XML reports...
2022-05-18T04:18:00.6660705Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20220518041800.xml
2022-05-18T04:18:01.3333011Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:18:01.3342460Z 
2022-05-18T04:18:01.3342566Z Running tests...
2022-05-18T04:18:01.3343518Z ----------------------------------------------------------------------
2022-05-18T04:18:01.3351177Z   test_nccl_blocking_wait_with_barrier (__main__.NcclErrorHandlingTest) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-05-18T04:18:01.3351522Z 
2022-05-18T04:18:01.3351843Z ----------------------------------------------------------------------
2022-05-18T04:18:01.3352126Z Ran 1 test in 0.001s
2022-05-18T04:18:01.3352244Z 
2022-05-18T04:18:01.3352307Z OK (skipped=1)
2022-05-18T04:18:01.3352415Z 
2022-05-18T04:18:01.3352554Z Generating XML reports...
2022-05-18T04:18:01.3377648Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20220518041801.xml
2022-05-18T04:18:02.0070628Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:18:02.0080763Z 
2022-05-18T04:18:02.0081032Z Running tests...
2022-05-18T04:18:02.0081669Z ----------------------------------------------------------------------
2022-05-18T04:18:02.0086870Z   test_nccl_errors_blocking_abort (__main__.NcclErrorHandlingTest) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-05-18T04:18:02.0087263Z 
2022-05-18T04:18:02.0087701Z ----------------------------------------------------------------------
2022-05-18T04:18:02.0088159Z Ran 1 test in 0.001s
2022-05-18T04:18:02.0088333Z 
2022-05-18T04:18:02.0088416Z OK (skipped=1)
2022-05-18T04:18:02.0088513Z 
2022-05-18T04:18:02.0088601Z Generating XML reports...
2022-05-18T04:18:02.0112248Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20220518041802.xml
2022-05-18T04:18:02.6827936Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:18:02.6837502Z 
2022-05-18T04:18:02.6837624Z Running tests...
2022-05-18T04:18:02.6837974Z ----------------------------------------------------------------------
2022-05-18T04:18:02.6843057Z   test_nccl_errors_blocking_clean_exit (__main__.NcclErrorHandlingTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-05-18T04:18:02.6843515Z 
2022-05-18T04:18:02.6843958Z ----------------------------------------------------------------------
2022-05-18T04:18:02.6844205Z Ran 1 test in 0.001s
2022-05-18T04:18:02.6844326Z 
2022-05-18T04:18:02.6844399Z OK (skipped=1)
2022-05-18T04:18:02.6844508Z 
2022-05-18T04:18:02.6844594Z Generating XML reports...
2022-05-18T04:18:02.6874581Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20220518041802.xml
2022-05-18T04:18:03.3549506Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:18:03.3558970Z 
2022-05-18T04:18:03.3559105Z Running tests...
2022-05-18T04:18:03.3559700Z ----------------------------------------------------------------------
2022-05-18T04:18:03.3564867Z   test_nccl_errors_blocking_nonzero_exit (__main__.NcclErrorHandlingTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-05-18T04:18:03.3565314Z 
2022-05-18T04:18:03.3565714Z ----------------------------------------------------------------------
2022-05-18T04:18:03.3566132Z Ran 1 test in 0.001s
2022-05-18T04:18:03.3566325Z 
2022-05-18T04:18:03.3566446Z OK (skipped=1)
2022-05-18T04:18:03.3566635Z 
2022-05-18T04:18:03.3566783Z Generating XML reports...
2022-05-18T04:18:03.3591019Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20220518041803.xml
2022-05-18T04:18:04.0277922Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:18:04.0287846Z 
2022-05-18T04:18:04.0288297Z Running tests...
2022-05-18T04:18:04.0288722Z ----------------------------------------------------------------------
2022-05-18T04:18:04.0293621Z   test_nccl_errors_blocking_sigkill (__main__.NcclErrorHandlingTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-05-18T04:18:04.0293991Z 
2022-05-18T04:18:04.0294361Z ----------------------------------------------------------------------
2022-05-18T04:18:04.0294653Z Ran 1 test in 0.001s
2022-05-18T04:18:04.0294769Z 
2022-05-18T04:18:04.0294849Z OK (skipped=1)
2022-05-18T04:18:04.0294959Z 
2022-05-18T04:18:04.0295031Z Generating XML reports...
2022-05-18T04:18:04.0320004Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20220518041804.xml
2022-05-18T04:18:04.7006031Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:18:04.7015568Z 
2022-05-18T04:18:04.7015669Z Running tests...
2022-05-18T04:18:04.7016092Z ----------------------------------------------------------------------
2022-05-18T04:18:04.7021814Z   test_nccl_errors_blocking_sigterm (__main__.NcclErrorHandlingTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-05-18T04:18:04.7022262Z 
2022-05-18T04:18:04.7022604Z ----------------------------------------------------------------------
2022-05-18T04:18:04.7022850Z Ran 1 test in 0.001s
2022-05-18T04:18:04.7023156Z 
2022-05-18T04:18:04.7023217Z OK (skipped=1)
2022-05-18T04:18:04.7023325Z 
2022-05-18T04:18:04.7023413Z Generating XML reports...
2022-05-18T04:18:04.7054930Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20220518041804.xml
2022-05-18T04:18:05.3740904Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:18:05.3750996Z 
2022-05-18T04:18:05.3751322Z Running tests...
2022-05-18T04:18:05.3752165Z ----------------------------------------------------------------------
2022-05-18T04:18:05.3763019Z   test_nccl_errors_nonblocking (__main__.NcclErrorHandlingTest) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-05-18T04:18:05.3763331Z 
2022-05-18T04:18:05.3763596Z ----------------------------------------------------------------------
2022-05-18T04:18:05.3763902Z Ran 1 test in 0.001s
2022-05-18T04:18:05.3764018Z 
2022-05-18T04:18:05.3764098Z OK (skipped=1)
2022-05-18T04:18:05.3764537Z 
2022-05-18T04:18:05.3764625Z Generating XML reports...
2022-05-18T04:18:05.3788256Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20220518041805.xml
2022-05-18T04:18:06.0437270Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:18:06.0446772Z 
2022-05-18T04:18:06.0446918Z Running tests...
2022-05-18T04:18:06.0447339Z ----------------------------------------------------------------------
2022-05-18T04:18:06.0458457Z   test_nccl_timeout (__main__.NcclErrorHandlingTest) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-05-18T04:18:06.0459287Z 
2022-05-18T04:18:06.0459734Z ----------------------------------------------------------------------
2022-05-18T04:18:06.0460036Z Ran 1 test in 0.001s
2022-05-18T04:18:06.0460150Z 
2022-05-18T04:18:06.0460225Z OK (skipped=1)
2022-05-18T04:18:06.0460378Z 
2022-05-18T04:18:06.0460455Z Generating XML reports...
2022-05-18T04:18:06.0483649Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20220518041806.xml
2022-05-18T04:18:06.7150935Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:18:06.7160538Z 
2022-05-18T04:18:06.7160675Z Running tests...
2022-05-18T04:18:06.7161244Z ----------------------------------------------------------------------
2022-05-18T04:18:06.7166858Z   test_init_no_gpus (__main__.ProcessGroupNCCLNoGPUTest) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-05-18T04:18:06.7167190Z 
2022-05-18T04:18:06.7167529Z ----------------------------------------------------------------------
2022-05-18T04:18:06.7167786Z Ran 1 test in 0.001s
2022-05-18T04:18:06.7167911Z 
2022-05-18T04:18:06.7167973Z OK (skipped=1)
2022-05-18T04:18:06.7168080Z 
2022-05-18T04:18:06.7168166Z Generating XML reports...
2022-05-18T04:18:06.7193346Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLNoGPUTest-20220518041806.xml
2022-05-18T04:18:07.3888656Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:18:07.3898200Z 
2022-05-18T04:18:07.3898333Z Running tests...
2022-05-18T04:18:07.3898760Z ----------------------------------------------------------------------
2022-05-18T04:18:07.3910796Z   test_allgather_base_basics (__main__.ProcessGroupNCCLTest) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-05-18T04:18:07.3911227Z 
2022-05-18T04:18:07.3911715Z ----------------------------------------------------------------------
2022-05-18T04:18:07.3911992Z Ran 1 test in 0.001s
2022-05-18T04:18:07.3912095Z 
2022-05-18T04:18:07.3912169Z OK (skipped=1)
2022-05-18T04:18:07.3912284Z 
2022-05-18T04:18:07.3912371Z Generating XML reports...
2022-05-18T04:18:07.3935824Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220518041807.xml
2022-05-18T04:18:08.0639055Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:18:08.0649102Z 
2022-05-18T04:18:08.0649214Z Running tests...
2022-05-18T04:18:08.0649687Z ----------------------------------------------------------------------
2022-05-18T04:18:08.0658907Z   test_allgather_base_ops (__main__.ProcessGroupNCCLTest) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-05-18T04:18:08.0659266Z 
2022-05-18T04:18:08.0659696Z ----------------------------------------------------------------------
2022-05-18T04:18:08.0660297Z Ran 1 test in 0.001s
2022-05-18T04:18:08.0660413Z 
2022-05-18T04:18:08.0660472Z OK (skipped=1)
2022-05-18T04:18:08.0660582Z 
2022-05-18T04:18:08.0660669Z Generating XML reports...
2022-05-18T04:18:08.0684212Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220518041808.xml
2022-05-18T04:18:08.7331707Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:18:08.7341664Z 
2022-05-18T04:18:08.7342119Z Running tests...
2022-05-18T04:18:08.7342501Z ----------------------------------------------------------------------
2022-05-18T04:18:08.7355133Z   test_allgather_ops (__main__.ProcessGroupNCCLTest) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-05-18T04:18:08.7355504Z 
2022-05-18T04:18:08.7355938Z ----------------------------------------------------------------------
2022-05-18T04:18:08.7356242Z Ran 1 test in 0.001s
2022-05-18T04:18:08.7356359Z 
2022-05-18T04:18:08.7356447Z OK (skipped=1)
2022-05-18T04:18:08.7356556Z 
2022-05-18T04:18:08.7356641Z Generating XML reports...
2022-05-18T04:18:08.7390254Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220518041808.xml
2022-05-18T04:18:09.4058831Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:18:09.4068571Z 
2022-05-18T04:18:09.4069040Z Running tests...
2022-05-18T04:18:09.4069685Z ----------------------------------------------------------------------
2022-05-18T04:18:09.4090526Z   test_allreduce_ops (__main__.ProcessGroupNCCLTest) ... skip: c10d was not compiled with the NCCL backend (0.002s)
2022-05-18T04:18:09.4090942Z 
2022-05-18T04:18:09.4091310Z ----------------------------------------------------------------------
2022-05-18T04:18:09.4091671Z Ran 1 test in 0.002s
2022-05-18T04:18:09.4091844Z 
2022-05-18T04:18:09.4091933Z OK (skipped=1)
2022-05-18T04:18:09.4092046Z 
2022-05-18T04:18:09.4092133Z Generating XML reports...
2022-05-18T04:18:09.4115583Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220518041809.xml
2022-05-18T04:18:10.0795435Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:18:10.0805673Z 
2022-05-18T04:18:10.0805803Z Running tests...
2022-05-18T04:18:10.0806299Z ----------------------------------------------------------------------
2022-05-18T04:18:10.0818497Z   test_barrier (__main__.ProcessGroupNCCLTest) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-05-18T04:18:10.0819085Z 
2022-05-18T04:18:10.0819739Z ----------------------------------------------------------------------
2022-05-18T04:18:10.0820086Z Ran 1 test in 0.001s
2022-05-18T04:18:10.0820192Z 
2022-05-18T04:18:10.0820279Z OK (skipped=1)
2022-05-18T04:18:10.0820389Z 
2022-05-18T04:18:10.0820477Z Generating XML reports...
2022-05-18T04:18:10.0844663Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220518041810.xml
2022-05-18T04:18:10.7504675Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:18:10.7514762Z 
2022-05-18T04:18:10.7515134Z Running tests...
2022-05-18T04:18:10.7515547Z ----------------------------------------------------------------------
2022-05-18T04:18:10.7529957Z   test_broadcast_ops (__main__.ProcessGroupNCCLTest) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-05-18T04:18:10.7530344Z 
2022-05-18T04:18:10.7530734Z ----------------------------------------------------------------------
2022-05-18T04:18:10.7531173Z Ran 1 test in 0.002s
2022-05-18T04:18:10.7531361Z 
2022-05-18T04:18:10.7531439Z OK (skipped=1)
2022-05-18T04:18:10.7531549Z 
2022-05-18T04:18:10.7531636Z Generating XML reports...
2022-05-18T04:18:10.7562732Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220518041810.xml
2022-05-18T04:18:11.4237732Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:18:11.4247679Z 
2022-05-18T04:18:11.4247841Z Running tests...
2022-05-18T04:18:11.4248517Z ----------------------------------------------------------------------
2022-05-18T04:18:11.4263786Z   test_empty_tensors (__main__.ProcessGroupNCCLTest) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-05-18T04:18:11.4264095Z 
2022-05-18T04:18:11.4264413Z ----------------------------------------------------------------------
2022-05-18T04:18:11.4264658Z Ran 1 test in 0.002s
2022-05-18T04:18:11.4264774Z 
2022-05-18T04:18:11.4264851Z OK (skipped=1)
2022-05-18T04:18:11.4264976Z 
2022-05-18T04:18:11.4265077Z Generating XML reports...
2022-05-18T04:18:11.4288502Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220518041811.xml
2022-05-18T04:18:12.0965529Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:18:12.0975082Z 
2022-05-18T04:18:12.0975217Z Running tests...
2022-05-18T04:18:12.0975800Z ----------------------------------------------------------------------
2022-05-18T04:18:12.0995036Z   test_gather_checks (__main__.ProcessGroupNCCLTest) ... skip: c10d was not compiled with the NCCL backend (0.002s)
2022-05-18T04:18:12.0995381Z 
2022-05-18T04:18:12.0995698Z ----------------------------------------------------------------------
2022-05-18T04:18:12.0996035Z Ran 1 test in 0.002s
2022-05-18T04:18:12.0996151Z 
2022-05-18T04:18:12.0996252Z OK (skipped=1)
2022-05-18T04:18:12.0996399Z 
2022-05-18T04:18:12.0996473Z Generating XML reports...
2022-05-18T04:18:12.1021057Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220518041812.xml
2022-05-18T04:18:12.7739470Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:18:12.7749766Z 
2022-05-18T04:18:12.7749878Z Running tests...
2022-05-18T04:18:12.7750462Z ----------------------------------------------------------------------
2022-05-18T04:18:12.7764186Z   test_gather_ops (__main__.ProcessGroupNCCLTest) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-05-18T04:18:12.7764567Z 
2022-05-18T04:18:12.7764995Z ----------------------------------------------------------------------
2022-05-18T04:18:12.7765449Z Ran 1 test in 0.001s
2022-05-18T04:18:12.7765632Z 
2022-05-18T04:18:12.7765727Z OK (skipped=1)
2022-05-18T04:18:12.7765836Z 
2022-05-18T04:18:12.7765925Z Generating XML reports...
2022-05-18T04:18:12.7797657Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220518041812.xml
2022-05-18T04:18:13.4479009Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:18:13.4489041Z 
2022-05-18T04:18:13.4489163Z Running tests...
2022-05-18T04:18:13.4490056Z ----------------------------------------------------------------------
2022-05-18T04:18:13.4506122Z   test_gather_stress (__main__.ProcessGroupNCCLTest) ... skip: c10d was not compiled with the NCCL backend (0.002s)
2022-05-18T04:18:13.4506484Z 
2022-05-18T04:18:13.4506937Z ----------------------------------------------------------------------
2022-05-18T04:18:13.4507378Z Ran 1 test in 0.002s
2022-05-18T04:18:13.4507581Z 
2022-05-18T04:18:13.4507642Z OK (skipped=1)
2022-05-18T04:18:13.4507749Z 
2022-05-18T04:18:13.4507862Z Generating XML reports...
2022-05-18T04:18:13.4531831Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220518041813.xml
2022-05-18T04:18:14.1196062Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:18:14.1205889Z 
2022-05-18T04:18:14.1206313Z Running tests...
2022-05-18T04:18:14.1206713Z ----------------------------------------------------------------------
2022-05-18T04:18:14.1220634Z   test_reduce_ops (__main__.ProcessGroupNCCLTest) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-05-18T04:18:14.1221316Z 
2022-05-18T04:18:14.1221560Z ----------------------------------------------------------------------
2022-05-18T04:18:14.1221809Z Ran 1 test in 0.002s
2022-05-18T04:18:14.1222010Z 
2022-05-18T04:18:14.1222071Z OK (skipped=1)
2022-05-18T04:18:14.1222177Z 
2022-05-18T04:18:14.1222263Z Generating XML reports...
2022-05-18T04:18:14.1245835Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220518041814.xml
2022-05-18T04:18:14.7899923Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:18:14.7909464Z 
2022-05-18T04:18:14.7909562Z Running tests...
2022-05-18T04:18:14.7910523Z ----------------------------------------------------------------------
2022-05-18T04:18:14.7922007Z   test_reduce_scatter_base_basics (__main__.ProcessGroupNCCLTest) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-05-18T04:18:14.7922528Z 
2022-05-18T04:18:14.7922838Z ----------------------------------------------------------------------
2022-05-18T04:18:14.7923102Z Ran 1 test in 0.001s
2022-05-18T04:18:14.7923217Z 
2022-05-18T04:18:14.7923291Z OK (skipped=1)
2022-05-18T04:18:14.7923400Z 
2022-05-18T04:18:14.7923478Z Generating XML reports...
2022-05-18T04:18:14.7954832Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220518041814.xml
2022-05-18T04:18:15.4670395Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:18:15.4680058Z 
2022-05-18T04:18:15.4680498Z Running tests...
2022-05-18T04:18:15.4681086Z ----------------------------------------------------------------------
2022-05-18T04:18:15.4689946Z   test_reduce_scatter_base_ops (__main__.ProcessGroupNCCLTest) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-05-18T04:18:15.4690218Z 
2022-05-18T04:18:15.4690587Z ----------------------------------------------------------------------
2022-05-18T04:18:15.4690852Z Ran 1 test in 0.001s
2022-05-18T04:18:15.4690987Z 
2022-05-18T04:18:15.4691106Z OK (skipped=1)
2022-05-18T04:18:15.4691235Z 
2022-05-18T04:18:15.4691368Z Generating XML reports...
2022-05-18T04:18:15.4715195Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220518041815.xml
2022-05-18T04:18:16.1403658Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:18:16.1412992Z 
2022-05-18T04:18:16.1413132Z Running tests...
2022-05-18T04:18:16.1413551Z ----------------------------------------------------------------------
2022-05-18T04:18:16.1443646Z   test_reduce_scatter_ops (__main__.ProcessGroupNCCLTest) ... skip: c10d was not compiled with the NCCL backend (0.003s)
2022-05-18T04:18:16.1444021Z 
2022-05-18T04:18:16.1444432Z ----------------------------------------------------------------------
2022-05-18T04:18:16.1444748Z Ran 1 test in 0.003s
2022-05-18T04:18:16.1444865Z 
2022-05-18T04:18:16.1444950Z OK (skipped=1)
2022-05-18T04:18:16.1445059Z 
2022-05-18T04:18:16.1445144Z Generating XML reports...
2022-05-18T04:18:16.1468638Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220518041816.xml
2022-05-18T04:18:16.8137550Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:18:16.8148077Z 
2022-05-18T04:18:16.8148257Z Running tests...
2022-05-18T04:18:16.8148654Z ----------------------------------------------------------------------
2022-05-18T04:18:16.8162506Z   test_scatter_checks (__main__.ProcessGroupNCCLTest) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-05-18T04:18:16.8163103Z 
2022-05-18T04:18:16.8163392Z ----------------------------------------------------------------------
2022-05-18T04:18:16.8163642Z Ran 1 test in 0.001s
2022-05-18T04:18:16.8163757Z 
2022-05-18T04:18:16.8163832Z OK (skipped=1)
2022-05-18T04:18:16.8163941Z 
2022-05-18T04:18:16.8164014Z Generating XML reports...
2022-05-18T04:18:16.8194511Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220518041816.xml
2022-05-18T04:18:17.4907958Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:18:17.4918234Z 
2022-05-18T04:18:17.4918610Z Running tests...
2022-05-18T04:18:17.4919254Z ----------------------------------------------------------------------
2022-05-18T04:18:17.4932850Z   test_scatter_ops (__main__.ProcessGroupNCCLTest) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-05-18T04:18:17.4933213Z 
2022-05-18T04:18:17.4933469Z ----------------------------------------------------------------------
2022-05-18T04:18:17.4933704Z Ran 1 test in 0.001s
2022-05-18T04:18:17.4933818Z 
2022-05-18T04:18:17.4933891Z OK (skipped=1)
2022-05-18T04:18:17.4933999Z 
2022-05-18T04:18:17.4934107Z Generating XML reports...
2022-05-18T04:18:17.4957955Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220518041817.xml
2022-05-18T04:18:18.1607312Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:18:18.1616319Z 
2022-05-18T04:18:18.1616463Z Running tests...
2022-05-18T04:18:18.1617044Z ----------------------------------------------------------------------
2022-05-18T04:18:18.1633773Z   test_scatter_stress (__main__.ProcessGroupNCCLTest) ... skip: c10d was not compiled with the NCCL backend (0.002s)
2022-05-18T04:18:18.1634161Z 
2022-05-18T04:18:18.1634533Z ----------------------------------------------------------------------
2022-05-18T04:18:18.1634961Z Ran 1 test in 0.002s
2022-05-18T04:18:18.1635120Z 
2022-05-18T04:18:18.1635195Z OK (skipped=1)
2022-05-18T04:18:18.1635303Z 
2022-05-18T04:18:18.1635389Z Generating XML reports...
2022-05-18T04:18:18.1659725Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220518041818.xml
2022-05-18T04:18:18.8134040Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:18:18.8143848Z 
2022-05-18T04:18:18.8144005Z Running tests...
2022-05-18T04:18:18.8144384Z ----------------------------------------------------------------------
2022-05-18T04:18:18.8178557Z   test_common_errors (__main__.RendezvousEnvTest) ... skip: c10d was not compiled with the NCCL backend (0.003s)
2022-05-18T04:18:18.8178845Z 
2022-05-18T04:18:18.8179153Z ----------------------------------------------------------------------
2022-05-18T04:18:18.8179383Z Ran 1 test in 0.003s
2022-05-18T04:18:18.8179496Z 
2022-05-18T04:18:18.8179569Z OK (skipped=1)
2022-05-18T04:18:18.8179676Z 
2022-05-18T04:18:18.8179765Z Generating XML reports...
2022-05-18T04:18:18.8203921Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-RendezvousEnvTest-20220518041818.xml
2022-05-18T04:18:19.4854546Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2022-05-18T04:18:19.4863769Z 
2022-05-18T04:18:19.4863911Z Running tests...
2022-05-18T04:18:19.4864505Z ----------------------------------------------------------------------
2022-05-18T04:18:19.4868727Z   test_default_store_timeout_nccl (__main__.TimeoutTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-05-18T04:18:19.4869237Z 
2022-05-18T04:18:19.4869812Z ----------------------------------------------------------------------
2022-05-18T04:18:19.4870258Z Ran 1 test in 0.001s
2022-05-18T04:18:19.4870445Z 
2022-05-18T04:18:19.4870583Z OK (skipped=1)
2022-05-18T04:18:19.4870766Z 
2022-05-18T04:18:19.4870892Z Generating XML reports...
2022-05-18T04:18:19.4895029Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-TimeoutTest-20220518041819.xml
2022-05-18T04:18:19.7027284Z Running distributed/test_c10d_spawn_gloo ... [2022-05-18 04:18:19.702364]
2022-05-18T04:18:19.7027903Z Executing ['/opt/conda/bin/python', 'distributed/test_c10d_spawn_gloo.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-05-18 04:18:19.702443]
2022-05-18T04:18:20.2596968Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4zf1_8f3
2022-05-18T04:18:20.2598010Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4zf1_8f3/_remote_module_non_scriptable.py
2022-05-18T04:18:20.5312402Z <unittest.suite.TestSuite tests=[<__main__.DistributedDataParallelSingleProcessTest testMethod=test_cpu>, <__main__.DistributedDataParallelSingleProcessTest testMethod=test_cuda>, <__main__.DistributedDataParallelSingleProcessTest testMethod=test_rnn>]>
2022-05-18T04:18:20.5313180Z test_cpu (__main__.DistributedDataParallelSingleProcessTest)
2022-05-18T04:18:20.5313524Z test_cuda (__main__.DistributedDataParallelSingleProcessTest)
2022-05-18T04:18:20.5313860Z test_rnn (__main__.DistributedDataParallelSingleProcessTest)
2022-05-18T04:18:20.5314529Z <unittest.suite.TestSuite tests=[<__main__.ProcessGroupShareTensorTest testMethod=test_shared_allgather_chunk_gloo>, <__main__.ProcessGroupShareTensorTest testMethod=test_shared_allgather_gloo>, <__main__.ProcessGroupShareTensorTest testMethod=test_shared_allreduce_gloo>, <__main__.ProcessGroupShareTensorTest testMethod=test_shared_broadcast_gloo>]>
2022-05-18T04:18:20.5315170Z test_shared_allgather_chunk_gloo (__main__.ProcessGroupShareTensorTest)
2022-05-18T04:18:20.5315490Z test_shared_allgather_gloo (__main__.ProcessGroupShareTensorTest)
2022-05-18T04:18:20.5315798Z test_shared_allreduce_gloo (__main__.ProcessGroupShareTensorTest)
2022-05-18T04:18:20.5316089Z test_shared_broadcast_gloo (__main__.ProcessGroupShareTensorTest)
2022-05-18T04:18:20.5316355Z <unittest.suite.TestSuite tests=[]>
2022-05-18T04:18:20.5316586Z <unittest.suite.TestSuite tests=[]>
2022-05-18T04:18:20.5317466Z <unittest.suite.TestSuite tests=[<__main__.TestDistributedNNFunctionsGloo testMethod=test_all_gather>, <__main__.TestDistributedNNFunctionsGloo testMethod=test_all_to_all>, <__main__.TestDistributedNNFunctionsGloo testMethod=test_all_to_all_single>, <__main__.TestDistributedNNFunctionsGloo testMethod=test_allreduce>, <__main__.TestDistributedNNFunctionsGloo testMethod=test_broadcast>, <__main__.TestDistributedNNFunctionsGloo testMethod=test_gather>, <__main__.TestDistributedNNFunctionsGloo testMethod=test_reduce>, <__main__.TestDistributedNNFunctionsGloo testMethod=test_scatter>]>
2022-05-18T04:18:20.5318352Z test_all_gather (__main__.TestDistributedNNFunctionsGloo)
2022-05-18T04:18:20.5318632Z test_all_to_all (__main__.TestDistributedNNFunctionsGloo)
2022-05-18T04:18:20.5318925Z test_all_to_all_single (__main__.TestDistributedNNFunctionsGloo)
2022-05-18T04:18:20.5319225Z test_allreduce (__main__.TestDistributedNNFunctionsGloo)
2022-05-18T04:18:20.5319501Z test_broadcast (__main__.TestDistributedNNFunctionsGloo)
2022-05-18T04:18:20.5319787Z test_gather (__main__.TestDistributedNNFunctionsGloo)
2022-05-18T04:18:20.5320068Z test_reduce (__main__.TestDistributedNNFunctionsGloo)
2022-05-18T04:18:20.5320350Z test_scatter (__main__.TestDistributedNNFunctionsGloo)
2022-05-18T04:18:21.0860776Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6bpb18nt
2022-05-18T04:18:21.0861930Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6bpb18nt/_remote_module_non_scriptable.py
2022-05-18T04:18:21.3553381Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo
2022-05-18T04:18:21.3562750Z 
2022-05-18T04:18:21.3563224Z Running tests...
2022-05-18T04:18:21.3563656Z ----------------------------------------------------------------------
2022-05-18T04:18:21.3611736Z   test_cpu (__main__.DistributedDataParallelSingleProcessTest) ... INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:18:21.3680471Z ok (0.012s)
2022-05-18T04:18:21.3681142Z 
2022-05-18T04:18:21.3681456Z ----------------------------------------------------------------------
2022-05-18T04:18:21.3681753Z Ran 1 test in 0.012s
2022-05-18T04:18:21.3681909Z 
2022-05-18T04:18:21.3682220Z OK
2022-05-18T04:18:21.3682315Z 
2022-05-18T04:18:21.3682412Z Generating XML reports...
2022-05-18T04:18:21.3710880Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-DistributedDataParallelSingleProcessTest-20220518041821.xml
2022-05-18T04:18:22.0790227Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuafqeham
2022-05-18T04:18:22.0790961Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuafqeham/_remote_module_non_scriptable.py
2022-05-18T04:18:22.3537951Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo
2022-05-18T04:18:22.3547192Z 
2022-05-18T04:18:22.3547303Z Running tests...
2022-05-18T04:18:22.3547794Z ----------------------------------------------------------------------
2022-05-18T04:18:22.3552681Z   test_cuda (__main__.DistributedDataParallelSingleProcessTest) ... skip: At least 1 CUDA GPUS needed (0.000s)
2022-05-18T04:18:22.3553118Z 
2022-05-18T04:18:22.3553557Z ----------------------------------------------------------------------
2022-05-18T04:18:22.3553982Z Ran 1 test in 0.001s
2022-05-18T04:18:22.3554167Z 
2022-05-18T04:18:22.3554284Z OK (skipped=1)
2022-05-18T04:18:22.3554459Z 
2022-05-18T04:18:22.3554593Z Generating XML reports...
2022-05-18T04:18:22.3583725Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-DistributedDataParallelSingleProcessTest-20220518041822.xml
2022-05-18T04:18:23.0659155Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv0rclwbh
2022-05-18T04:18:23.0659871Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv0rclwbh/_remote_module_non_scriptable.py
2022-05-18T04:18:23.3342472Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo
2022-05-18T04:18:23.3352114Z 
2022-05-18T04:18:23.3352432Z Running tests...
2022-05-18T04:18:23.3353034Z ----------------------------------------------------------------------
2022-05-18T04:18:23.3365367Z   test_rnn (__main__.DistributedDataParallelSingleProcessTest) ... skip: At least 1 CUDA GPUS needed (0.001s)
2022-05-18T04:18:23.3365814Z 
2022-05-18T04:18:23.3366266Z ----------------------------------------------------------------------
2022-05-18T04:18:23.3366517Z Ran 1 test in 0.001s
2022-05-18T04:18:23.3366619Z 
2022-05-18T04:18:23.3366693Z OK (skipped=1)
2022-05-18T04:18:23.3366801Z 
2022-05-18T04:18:23.3366885Z Generating XML reports...
2022-05-18T04:18:23.3396410Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-DistributedDataParallelSingleProcessTest-20220518041823.xml
2022-05-18T04:18:24.0443019Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd81pa8h6
2022-05-18T04:18:24.0443881Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd81pa8h6/_remote_module_non_scriptable.py
2022-05-18T04:18:24.3135546Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo
2022-05-18T04:18:24.3145549Z 
2022-05-18T04:18:24.3145648Z Running tests...
2022-05-18T04:18:24.3146357Z ----------------------------------------------------------------------
2022-05-18T04:18:24.3151123Z   test_shared_allgather_chunk_gloo (__main__.ProcessGroupShareTensorTest) ... skip: At least 2 CUDA GPUS needed (0.000s)
2022-05-18T04:18:24.3151502Z 
2022-05-18T04:18:24.3151882Z ----------------------------------------------------------------------
2022-05-18T04:18:24.3152333Z Ran 1 test in 0.001s
2022-05-18T04:18:24.3152466Z 
2022-05-18T04:18:24.3152542Z OK (skipped=1)
2022-05-18T04:18:24.3152649Z 
2022-05-18T04:18:24.3152725Z Generating XML reports...
2022-05-18T04:18:24.3180466Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-ProcessGroupShareTensorTest-20220518041824.xml
2022-05-18T04:18:25.0225655Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmghw24go
2022-05-18T04:18:25.0226206Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmghw24go/_remote_module_non_scriptable.py
2022-05-18T04:18:25.2954569Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo
2022-05-18T04:18:25.2963970Z 
2022-05-18T04:18:25.2964248Z Running tests...
2022-05-18T04:18:25.2964682Z ----------------------------------------------------------------------
2022-05-18T04:18:25.2969864Z   test_shared_allgather_gloo (__main__.ProcessGroupShareTensorTest) ... skip: At least 2 CUDA GPUS needed (0.000s)
2022-05-18T04:18:25.2970250Z 
2022-05-18T04:18:25.2970668Z ----------------------------------------------------------------------
2022-05-18T04:18:25.2971133Z Ran 1 test in 0.001s
2022-05-18T04:18:25.2971338Z 
2022-05-18T04:18:25.2971414Z OK (skipped=1)
2022-05-18T04:18:25.2971523Z 
2022-05-18T04:18:25.2971612Z Generating XML reports...
2022-05-18T04:18:25.3000829Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-ProcessGroupShareTensorTest-20220518041825.xml
2022-05-18T04:18:26.0166012Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5fw_jx9s
2022-05-18T04:18:26.0166567Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5fw_jx9s/_remote_module_non_scriptable.py
2022-05-18T04:18:26.2849442Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo
2022-05-18T04:18:26.2858899Z 
2022-05-18T04:18:26.2858994Z Running tests...
2022-05-18T04:18:26.2859461Z ----------------------------------------------------------------------
2022-05-18T04:18:26.2865046Z   test_shared_allreduce_gloo (__main__.ProcessGroupShareTensorTest) ... skip: At least 2 CUDA GPUS needed (0.000s)
2022-05-18T04:18:26.2865408Z 
2022-05-18T04:18:26.2865684Z ----------------------------------------------------------------------
2022-05-18T04:18:26.2865934Z Ran 1 test in 0.001s
2022-05-18T04:18:26.2866052Z 
2022-05-18T04:18:26.2866111Z OK (skipped=1)
2022-05-18T04:18:26.2866225Z 
2022-05-18T04:18:26.2866311Z Generating XML reports...
2022-05-18T04:18:26.2896613Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-ProcessGroupShareTensorTest-20220518041826.xml
2022-05-18T04:18:26.9943404Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpklxxn2en
2022-05-18T04:18:26.9944553Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpklxxn2en/_remote_module_non_scriptable.py
2022-05-18T04:18:27.2621488Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo
2022-05-18T04:18:27.2630816Z 
2022-05-18T04:18:27.2630915Z Running tests...
2022-05-18T04:18:27.2631495Z ----------------------------------------------------------------------
2022-05-18T04:18:27.2636792Z   test_shared_broadcast_gloo (__main__.ProcessGroupShareTensorTest) ... skip: At least 2 CUDA GPUS needed (0.000s)
2022-05-18T04:18:27.2637185Z 
2022-05-18T04:18:27.2637572Z ----------------------------------------------------------------------
2022-05-18T04:18:27.2637920Z Ran 1 test in 0.001s
2022-05-18T04:18:27.2638057Z 
2022-05-18T04:18:27.2638162Z OK (skipped=1)
2022-05-18T04:18:27.2638311Z 
2022-05-18T04:18:27.2638415Z Generating XML reports...
2022-05-18T04:18:27.2667247Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-ProcessGroupShareTensorTest-20220518041827.xml
2022-05-18T04:18:27.9852185Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9cq552_1
2022-05-18T04:18:27.9852872Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9cq552_1/_remote_module_non_scriptable.py
2022-05-18T04:18:28.2557863Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo
2022-05-18T04:18:28.2567831Z 
2022-05-18T04:18:28.2568094Z Running tests...
2022-05-18T04:18:28.2568724Z ----------------------------------------------------------------------
2022-05-18T04:18:28.2726535Z   test_all_gather (__main__.TestDistributedNNFunctionsGloo) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10324
2022-05-18T04:18:28.2748483Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10325
2022-05-18T04:18:28.8323141Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptpn6jdvm
2022-05-18T04:18:28.8323876Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptpn6jdvm/_remote_module_non_scriptable.py
2022-05-18T04:18:28.8331290Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjfiblu07
2022-05-18T04:18:28.8333387Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjfiblu07/_remote_module_non_scriptable.py
2022-05-18T04:18:29.1045026Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:18:29.1068378Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:18:29.3777171Z skip: Need at least 2 CUDA devices (1.121s)
2022-05-18T04:18:29.3777449Z 
2022-05-18T04:18:29.3777921Z ----------------------------------------------------------------------
2022-05-18T04:18:29.3778350Z Ran 1 test in 1.121s
2022-05-18T04:18:29.3778538Z 
2022-05-18T04:18:29.3778649Z OK (skipped=1)
2022-05-18T04:18:29.3778820Z 
2022-05-18T04:18:29.3778949Z Generating XML reports...
2022-05-18T04:18:29.3813974Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20220518041828.xml
2022-05-18T04:18:30.1219483Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp88v4fl6k
2022-05-18T04:18:30.1220357Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp88v4fl6k/_remote_module_non_scriptable.py
2022-05-18T04:18:30.3894207Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo
2022-05-18T04:18:30.3903356Z 
2022-05-18T04:18:30.3903755Z Running tests...
2022-05-18T04:18:30.3904370Z ----------------------------------------------------------------------
2022-05-18T04:18:30.4059939Z   test_all_to_all (__main__.TestDistributedNNFunctionsGloo) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10357
2022-05-18T04:18:30.4081769Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10358
2022-05-18T04:18:30.9582595Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_p0hwm8o
2022-05-18T04:18:30.9583907Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_p0hwm8o/_remote_module_non_scriptable.py
2022-05-18T04:18:30.9668694Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjt5dshf4
2022-05-18T04:18:30.9669841Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjt5dshf4/_remote_module_non_scriptable.py
2022-05-18T04:18:31.2332138Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:18:31.2414300Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:18:31.5111409Z skip: Need at least 2 CUDA devices (1.120s)
2022-05-18T04:18:31.5111724Z 
2022-05-18T04:18:31.5112240Z ----------------------------------------------------------------------
2022-05-18T04:18:31.5112509Z Ran 1 test in 1.121s
2022-05-18T04:18:31.5112631Z 
2022-05-18T04:18:31.5112709Z OK (skipped=1)
2022-05-18T04:18:31.5112804Z 
2022-05-18T04:18:31.5112891Z Generating XML reports...
2022-05-18T04:18:31.5146461Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20220518041830.xml
2022-05-18T04:18:32.2523200Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl936kicf
2022-05-18T04:18:32.2523877Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl936kicf/_remote_module_non_scriptable.py
2022-05-18T04:18:32.5235839Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo
2022-05-18T04:18:32.5245650Z 
2022-05-18T04:18:32.5245786Z Running tests...
2022-05-18T04:18:32.5246399Z ----------------------------------------------------------------------
2022-05-18T04:18:32.5404920Z   test_all_to_all_single (__main__.TestDistributedNNFunctionsGloo) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10390
2022-05-18T04:18:32.5426706Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10391
2022-05-18T04:18:33.0945361Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpni_eu2h_
2022-05-18T04:18:33.0946178Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp37d2mnf6
2022-05-18T04:18:33.0946886Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpni_eu2h_/_remote_module_non_scriptable.py
2022-05-18T04:18:33.0947658Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp37d2mnf6/_remote_module_non_scriptable.py
2022-05-18T04:18:33.3661101Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:18:33.3667159Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:18:33.6455541Z skip: Need at least 2 CUDA devices (1.121s)
2022-05-18T04:18:33.6455805Z 
2022-05-18T04:18:33.6456172Z ----------------------------------------------------------------------
2022-05-18T04:18:33.6456424Z Ran 1 test in 1.121s
2022-05-18T04:18:33.6456525Z 
2022-05-18T04:18:33.6456601Z OK (skipped=1)
2022-05-18T04:18:33.6456709Z 
2022-05-18T04:18:33.6456796Z Generating XML reports...
2022-05-18T04:18:33.6491207Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20220518041832.xml
2022-05-18T04:18:34.3874743Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2bxzaq3_
2022-05-18T04:18:34.3875484Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2bxzaq3_/_remote_module_non_scriptable.py
2022-05-18T04:18:34.6570925Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo
2022-05-18T04:18:34.6580478Z 
2022-05-18T04:18:34.6580861Z Running tests...
2022-05-18T04:18:34.6581287Z ----------------------------------------------------------------------
2022-05-18T04:18:34.6737048Z   test_allreduce (__main__.TestDistributedNNFunctionsGloo) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10423
2022-05-18T04:18:34.6758699Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10424
2022-05-18T04:18:35.2396647Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc9oihpxf
2022-05-18T04:18:35.2397428Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc9oihpxf/_remote_module_non_scriptable.py
2022-05-18T04:18:35.2575954Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp50dcdpfw
2022-05-18T04:18:35.2577918Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp50dcdpfw/_remote_module_non_scriptable.py
2022-05-18T04:18:35.5089574Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:18:35.5286280Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:18:35.7788499Z skip: Need at least 2 CUDA devices (1.121s)
2022-05-18T04:18:35.7789074Z 
2022-05-18T04:18:35.7789661Z ----------------------------------------------------------------------
2022-05-18T04:18:35.7790143Z Ran 1 test in 1.121s
2022-05-18T04:18:35.7790355Z 
2022-05-18T04:18:35.7790465Z OK (skipped=1)
2022-05-18T04:18:35.7790575Z 
2022-05-18T04:18:35.7790650Z Generating XML reports...
2022-05-18T04:18:35.7824302Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20220518041834.xml
2022-05-18T04:18:36.5248639Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi2xsmewp
2022-05-18T04:18:36.5249704Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi2xsmewp/_remote_module_non_scriptable.py
2022-05-18T04:18:36.7943887Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo
2022-05-18T04:18:36.7953501Z 
2022-05-18T04:18:36.7953799Z Running tests...
2022-05-18T04:18:36.7954654Z ----------------------------------------------------------------------
2022-05-18T04:18:36.8112106Z   test_broadcast (__main__.TestDistributedNNFunctionsGloo) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10456
2022-05-18T04:18:36.8134319Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10457
2022-05-18T04:18:37.3793985Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmind25ve
2022-05-18T04:18:37.3794970Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmind25ve/_remote_module_non_scriptable.py
2022-05-18T04:18:37.4132661Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgqx4c_h1
2022-05-18T04:18:37.4134168Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgqx4c_h1/_remote_module_non_scriptable.py
2022-05-18T04:18:37.6506733Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:18:37.6849860Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:18:37.9163916Z skip: Need at least 2 CUDA devices (1.121s)
2022-05-18T04:18:37.9164217Z 
2022-05-18T04:18:37.9164680Z ----------------------------------------------------------------------
2022-05-18T04:18:37.9164932Z Ran 1 test in 1.121s
2022-05-18T04:18:37.9165048Z 
2022-05-18T04:18:37.9165122Z OK (skipped=1)
2022-05-18T04:18:37.9165234Z 
2022-05-18T04:18:37.9165307Z Generating XML reports...
2022-05-18T04:18:37.9198997Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20220518041836.xml
2022-05-18T04:18:38.6533063Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr_716e_w
2022-05-18T04:18:38.6533765Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr_716e_w/_remote_module_non_scriptable.py
2022-05-18T04:18:38.9236089Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo
2022-05-18T04:18:38.9246201Z 
2022-05-18T04:18:38.9246651Z Running tests...
2022-05-18T04:18:38.9247124Z ----------------------------------------------------------------------
2022-05-18T04:18:38.9411806Z   test_gather (__main__.TestDistributedNNFunctionsGloo) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10489
2022-05-18T04:18:38.9434228Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10490
2022-05-18T04:18:39.5074042Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmvr14hj5
2022-05-18T04:18:39.5074514Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmvr14hj5/_remote_module_non_scriptable.py
2022-05-18T04:18:39.5120384Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpytxqvuzr
2022-05-18T04:18:39.5122409Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpytxqvuzr/_remote_module_non_scriptable.py
2022-05-18T04:18:39.7787105Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:18:39.7817156Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:18:40.0462138Z skip: Need at least 2 CUDA devices (1.121s)
2022-05-18T04:18:40.0462434Z 
2022-05-18T04:18:40.0463127Z ----------------------------------------------------------------------
2022-05-18T04:18:40.0463485Z Ran 1 test in 1.122s
2022-05-18T04:18:40.0463602Z 
2022-05-18T04:18:40.0463679Z OK (skipped=1)
2022-05-18T04:18:40.0463789Z 
2022-05-18T04:18:40.0463877Z Generating XML reports...
2022-05-18T04:18:40.0497939Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20220518041838.xml
2022-05-18T04:18:40.7998956Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpojpw8mmz
2022-05-18T04:18:40.7999902Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpojpw8mmz/_remote_module_non_scriptable.py
2022-05-18T04:18:41.0682440Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo
2022-05-18T04:18:41.0691693Z 
2022-05-18T04:18:41.0692295Z Running tests...
2022-05-18T04:18:41.0692705Z ----------------------------------------------------------------------
2022-05-18T04:18:41.0848109Z   test_reduce (__main__.TestDistributedNNFunctionsGloo) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10522
2022-05-18T04:18:41.0869606Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10523
2022-05-18T04:18:41.6662298Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpiexh3hfw
2022-05-18T04:18:41.6663176Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpiexh3hfw/_remote_module_non_scriptable.py
2022-05-18T04:18:41.6663857Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphbw5nuvl
2022-05-18T04:18:41.6665295Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphbw5nuvl/_remote_module_non_scriptable.py
2022-05-18T04:18:41.9376546Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:18:41.9399831Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:18:42.1898729Z skip: Need at least 2 CUDA devices (1.120s)
2022-05-18T04:18:42.1898948Z 
2022-05-18T04:18:42.1899333Z ----------------------------------------------------------------------
2022-05-18T04:18:42.1899587Z Ran 1 test in 1.121s
2022-05-18T04:18:42.1899705Z 
2022-05-18T04:18:42.1899781Z OK (skipped=1)
2022-05-18T04:18:42.1899877Z 
2022-05-18T04:18:42.1899971Z Generating XML reports...
2022-05-18T04:18:42.1934088Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20220518041841.xml
2022-05-18T04:18:42.9384344Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsejqrtjt
2022-05-18T04:18:42.9385082Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsejqrtjt/_remote_module_non_scriptable.py
2022-05-18T04:18:43.2057100Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo
2022-05-18T04:18:43.2067409Z 
2022-05-18T04:18:43.2067666Z Running tests...
2022-05-18T04:18:43.2068328Z ----------------------------------------------------------------------
2022-05-18T04:18:43.2238031Z   test_scatter (__main__.TestDistributedNNFunctionsGloo) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10555
2022-05-18T04:18:43.2260718Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10556
2022-05-18T04:18:43.7843622Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa05ecovo
2022-05-18T04:18:43.7844420Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa05ecovo/_remote_module_non_scriptable.py
2022-05-18T04:18:43.8097917Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpej7fmmab
2022-05-18T04:18:43.8099207Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpej7fmmab/_remote_module_non_scriptable.py
2022-05-18T04:18:44.0566694Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:18:44.0789570Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:18:44.3290572Z skip: Need at least 2 CUDA devices (1.122s)
2022-05-18T04:18:44.3290833Z 
2022-05-18T04:18:44.3291358Z ----------------------------------------------------------------------
2022-05-18T04:18:44.3291679Z Ran 1 test in 1.122s
2022-05-18T04:18:44.3291795Z 
2022-05-18T04:18:44.3291869Z OK (skipped=1)
2022-05-18T04:18:44.3291977Z 
2022-05-18T04:18:44.3292066Z Generating XML reports...
2022-05-18T04:18:44.3325832Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20220518041843.xml
2022-05-18T04:18:44.6912885Z Running distributed/test_c10d_spawn_nccl ... [2022-05-18 04:18:44.690898]
2022-05-18T04:18:44.6913471Z Executing ['/opt/conda/bin/python', 'distributed/test_c10d_spawn_nccl.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-05-18 04:18:44.690982]
2022-05-18T04:18:45.2408505Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6gl128m7
2022-05-18T04:18:45.2409298Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6gl128m7/_remote_module_non_scriptable.py
2022-05-18T04:18:45.5090072Z <unittest.suite.TestSuite tests=[<__main__.ProcessGroupShareTensorTest testMethod=test_shared_allgather_nccl>, <__main__.ProcessGroupShareTensorTest testMethod=test_shared_allreduce_nccl>, <__main__.ProcessGroupShareTensorTest testMethod=test_shared_broadcast_nccl>, <__main__.ProcessGroupShareTensorTest testMethod=test_shared_reduce_nccl>]>
2022-05-18T04:18:45.5091333Z test_shared_allgather_nccl (__main__.ProcessGroupShareTensorTest)
2022-05-18T04:18:45.5091858Z test_shared_allreduce_nccl (__main__.ProcessGroupShareTensorTest)
2022-05-18T04:18:45.5092192Z test_shared_broadcast_nccl (__main__.ProcessGroupShareTensorTest)
2022-05-18T04:18:45.5092498Z test_shared_reduce_nccl (__main__.ProcessGroupShareTensorTest)
2022-05-18T04:18:45.5092756Z <unittest.suite.TestSuite tests=[]>
2022-05-18T04:18:45.5092992Z <unittest.suite.TestSuite tests=[]>
2022-05-18T04:18:45.5093820Z <unittest.suite.TestSuite tests=[<__main__.TestDistributedNNFunctionsNccl testMethod=test_all_gather>, <__main__.TestDistributedNNFunctionsNccl testMethod=test_all_to_all>, <__main__.TestDistributedNNFunctionsNccl testMethod=test_all_to_all_single>, <__main__.TestDistributedNNFunctionsNccl testMethod=test_allreduce>, <__main__.TestDistributedNNFunctionsNccl testMethod=test_broadcast>, <__main__.TestDistributedNNFunctionsNccl testMethod=test_reduce>, <__main__.TestDistributedNNFunctionsNccl testMethod=test_reduce_scatter>]>
2022-05-18T04:18:45.5094641Z test_all_gather (__main__.TestDistributedNNFunctionsNccl)
2022-05-18T04:18:45.5094926Z test_all_to_all (__main__.TestDistributedNNFunctionsNccl)
2022-05-18T04:18:45.5095222Z test_all_to_all_single (__main__.TestDistributedNNFunctionsNccl)
2022-05-18T04:18:45.5095526Z test_allreduce (__main__.TestDistributedNNFunctionsNccl)
2022-05-18T04:18:45.5095803Z test_broadcast (__main__.TestDistributedNNFunctionsNccl)
2022-05-18T04:18:45.5096095Z test_reduce (__main__.TestDistributedNNFunctionsNccl)
2022-05-18T04:18:45.5096388Z test_reduce_scatter (__main__.TestDistributedNNFunctionsNccl)
2022-05-18T04:18:46.0599042Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu3svmpgv
2022-05-18T04:18:46.0599510Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu3svmpgv/_remote_module_non_scriptable.py
2022-05-18T04:18:46.3322205Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_nccl
2022-05-18T04:18:46.3332558Z 
2022-05-18T04:18:46.3332936Z Running tests...
2022-05-18T04:18:46.3333369Z ----------------------------------------------------------------------
2022-05-18T04:18:46.3338137Z   test_shared_allgather_nccl (__main__.ProcessGroupShareTensorTest) ... skip: At least 2 CUDA GPUS needed (0.000s)
2022-05-18T04:18:46.3338565Z 
2022-05-18T04:18:46.3338856Z ----------------------------------------------------------------------
2022-05-18T04:18:46.3339115Z Ran 1 test in 0.001s
2022-05-18T04:18:46.3339287Z 
2022-05-18T04:18:46.3339366Z OK (skipped=1)
2022-05-18T04:18:46.3339474Z 
2022-05-18T04:18:46.3339547Z Generating XML reports...
2022-05-18T04:18:46.3368228Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-ProcessGroupShareTensorTest-20220518041846.xml
2022-05-18T04:18:47.0410173Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp47m4x6hl
2022-05-18T04:18:47.0411376Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp47m4x6hl/_remote_module_non_scriptable.py
2022-05-18T04:18:47.3102147Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_nccl
2022-05-18T04:18:47.3111836Z 
2022-05-18T04:18:47.3111983Z Running tests...
2022-05-18T04:18:47.3112819Z ----------------------------------------------------------------------
2022-05-18T04:18:47.3117675Z   test_shared_allreduce_nccl (__main__.ProcessGroupShareTensorTest) ... skip: At least 2 CUDA GPUS needed (0.000s)
2022-05-18T04:18:47.3117926Z 
2022-05-18T04:18:47.3118446Z ----------------------------------------------------------------------
2022-05-18T04:18:47.3118729Z Ran 1 test in 0.001s
2022-05-18T04:18:47.3118832Z 
2022-05-18T04:18:47.3118915Z OK (skipped=1)
2022-05-18T04:18:47.3119022Z 
2022-05-18T04:18:47.3119109Z Generating XML reports...
2022-05-18T04:18:47.3149222Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-ProcessGroupShareTensorTest-20220518041847.xml
2022-05-18T04:18:48.0327402Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_btppqts
2022-05-18T04:18:48.0328131Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_btppqts/_remote_module_non_scriptable.py
2022-05-18T04:18:48.3005950Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_nccl
2022-05-18T04:18:48.3015365Z 
2022-05-18T04:18:48.3015633Z Running tests...
2022-05-18T04:18:48.3016006Z ----------------------------------------------------------------------
2022-05-18T04:18:48.3020972Z   test_shared_broadcast_nccl (__main__.ProcessGroupShareTensorTest) ... skip: At least 2 CUDA GPUS needed (0.000s)
2022-05-18T04:18:48.3021320Z 
2022-05-18T04:18:48.3021584Z ----------------------------------------------------------------------
2022-05-18T04:18:48.3021926Z Ran 1 test in 0.001s
2022-05-18T04:18:48.3022131Z 
2022-05-18T04:18:48.3022268Z OK (skipped=1)
2022-05-18T04:18:48.3022473Z 
2022-05-18T04:18:48.3022650Z Generating XML reports...
2022-05-18T04:18:48.3051113Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-ProcessGroupShareTensorTest-20220518041848.xml
2022-05-18T04:18:49.0072448Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmwjodg1_
2022-05-18T04:18:49.0073319Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmwjodg1_/_remote_module_non_scriptable.py
2022-05-18T04:18:49.2757263Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_nccl
2022-05-18T04:18:49.2766918Z 
2022-05-18T04:18:49.2767019Z Running tests...
2022-05-18T04:18:49.2767470Z ----------------------------------------------------------------------
2022-05-18T04:18:49.2773233Z   test_shared_reduce_nccl (__main__.ProcessGroupShareTensorTest) ... skip: At least 2 CUDA GPUS needed (0.000s)
2022-05-18T04:18:49.2773653Z 
2022-05-18T04:18:49.2774074Z ----------------------------------------------------------------------
2022-05-18T04:18:49.2774482Z Ran 1 test in 0.001s
2022-05-18T04:18:49.2774667Z 
2022-05-18T04:18:49.2774786Z OK (skipped=1)
2022-05-18T04:18:49.2774974Z 
2022-05-18T04:18:49.2775117Z Generating XML reports...
2022-05-18T04:18:49.2804394Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-ProcessGroupShareTensorTest-20220518041849.xml
2022-05-18T04:18:49.9960953Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu3ewb826
2022-05-18T04:18:49.9961786Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu3ewb826/_remote_module_non_scriptable.py
2022-05-18T04:18:50.2653204Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_nccl
2022-05-18T04:18:50.2664910Z 
2022-05-18T04:18:50.2665186Z Running tests...
2022-05-18T04:18:50.2665789Z ----------------------------------------------------------------------
2022-05-18T04:18:50.2668453Z   test_all_gather (__main__.TestDistributedNNFunctionsNccl) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-05-18T04:18:50.2669187Z 
2022-05-18T04:18:50.2669627Z ----------------------------------------------------------------------
2022-05-18T04:18:50.2670171Z Ran 1 test in 0.001s
2022-05-18T04:18:50.2670378Z 
2022-05-18T04:18:50.2670479Z OK (skipped=1)
2022-05-18T04:18:50.2670590Z 
2022-05-18T04:18:50.2670677Z Generating XML reports...
2022-05-18T04:18:50.2700335Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20220518041850.xml
2022-05-18T04:18:50.9733895Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqgo1bfkk
2022-05-18T04:18:50.9734630Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqgo1bfkk/_remote_module_non_scriptable.py
2022-05-18T04:18:51.2430989Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_nccl
2022-05-18T04:18:51.2440831Z 
2022-05-18T04:18:51.2441291Z Running tests...
2022-05-18T04:18:51.2441772Z ----------------------------------------------------------------------
2022-05-18T04:18:51.2445092Z   test_all_to_all (__main__.TestDistributedNNFunctionsNccl) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-05-18T04:18:51.2445429Z 
2022-05-18T04:18:51.2445714Z ----------------------------------------------------------------------
2022-05-18T04:18:51.2445949Z Ran 1 test in 0.000s
2022-05-18T04:18:51.2446068Z 
2022-05-18T04:18:51.2446141Z OK (skipped=1)
2022-05-18T04:18:51.2446250Z 
2022-05-18T04:18:51.2446342Z Generating XML reports...
2022-05-18T04:18:51.2475078Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20220518041851.xml
2022-05-18T04:18:51.9558878Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi10qu6mc
2022-05-18T04:18:51.9559796Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi10qu6mc/_remote_module_non_scriptable.py
2022-05-18T04:18:52.2222374Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_nccl
2022-05-18T04:18:52.2232294Z 
2022-05-18T04:18:52.2232411Z Running tests...
2022-05-18T04:18:52.2233009Z ----------------------------------------------------------------------
2022-05-18T04:18:52.2237360Z   test_all_to_all_single (__main__.TestDistributedNNFunctionsNccl) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-05-18T04:18:52.2237750Z 
2022-05-18T04:18:52.2238014Z ----------------------------------------------------------------------
2022-05-18T04:18:52.2238327Z Ran 1 test in 0.000s
2022-05-18T04:18:52.2238444Z 
2022-05-18T04:18:52.2238518Z OK (skipped=1)
2022-05-18T04:18:52.2238614Z 
2022-05-18T04:18:52.2238701Z Generating XML reports...
2022-05-18T04:18:52.2267122Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20220518041852.xml
2022-05-18T04:18:52.9483000Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp584y926k
2022-05-18T04:18:52.9484098Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp584y926k/_remote_module_non_scriptable.py
2022-05-18T04:18:53.2146329Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_nccl
2022-05-18T04:18:53.2156674Z 
2022-05-18T04:18:53.2156972Z Running tests...
2022-05-18T04:18:53.2157434Z ----------------------------------------------------------------------
2022-05-18T04:18:53.2161255Z   test_allreduce (__main__.TestDistributedNNFunctionsNccl) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-05-18T04:18:53.2161742Z 
2022-05-18T04:18:53.2162231Z ----------------------------------------------------------------------
2022-05-18T04:18:53.2162533Z Ran 1 test in 0.000s
2022-05-18T04:18:53.2162649Z 
2022-05-18T04:18:53.2162805Z OK (skipped=1)
2022-05-18T04:18:53.2162917Z 
2022-05-18T04:18:53.2163005Z Generating XML reports...
2022-05-18T04:18:53.2191286Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20220518041853.xml
2022-05-18T04:18:53.9303267Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5ztlhqbo
2022-05-18T04:18:53.9304256Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5ztlhqbo/_remote_module_non_scriptable.py
2022-05-18T04:18:54.1982014Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_nccl
2022-05-18T04:18:54.1992382Z 
2022-05-18T04:18:54.1992489Z Running tests...
2022-05-18T04:18:54.1993068Z ----------------------------------------------------------------------
2022-05-18T04:18:54.1997799Z   test_broadcast (__main__.TestDistributedNNFunctionsNccl) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-05-18T04:18:54.1998238Z 
2022-05-18T04:18:54.1998701Z ----------------------------------------------------------------------
2022-05-18T04:18:54.1999113Z Ran 1 test in 0.000s
2022-05-18T04:18:54.1999227Z 
2022-05-18T04:18:54.1999315Z OK (skipped=1)
2022-05-18T04:18:54.1999425Z 
2022-05-18T04:18:54.1999496Z Generating XML reports...
2022-05-18T04:18:54.2027838Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20220518041854.xml
2022-05-18T04:18:54.9053162Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprensetey
2022-05-18T04:18:54.9054204Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprensetey/_remote_module_non_scriptable.py
2022-05-18T04:18:55.1732094Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_nccl
2022-05-18T04:18:55.1742046Z 
2022-05-18T04:18:55.1742144Z Running tests...
2022-05-18T04:18:55.1742709Z ----------------------------------------------------------------------
2022-05-18T04:18:55.1746905Z   test_reduce (__main__.TestDistributedNNFunctionsNccl) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-05-18T04:18:55.1747446Z 
2022-05-18T04:18:55.1747876Z ----------------------------------------------------------------------
2022-05-18T04:18:55.1748196Z Ran 1 test in 0.000s
2022-05-18T04:18:55.1748313Z 
2022-05-18T04:18:55.1748385Z OK (skipped=1)
2022-05-18T04:18:55.1748496Z 
2022-05-18T04:18:55.1748581Z Generating XML reports...
2022-05-18T04:18:55.1776815Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20220518041855.xml
2022-05-18T04:18:55.8985657Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa177j5wf
2022-05-18T04:18:55.8986350Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa177j5wf/_remote_module_non_scriptable.py
2022-05-18T04:18:56.1696930Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_nccl
2022-05-18T04:18:56.1707040Z 
2022-05-18T04:18:56.1707168Z Running tests...
2022-05-18T04:18:56.1707579Z ----------------------------------------------------------------------
2022-05-18T04:18:56.1722926Z   test_reduce_scatter (__main__.TestDistributedNNFunctionsNccl) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-05-18T04:18:56.1723357Z 
2022-05-18T04:18:56.1723769Z ----------------------------------------------------------------------
2022-05-18T04:18:56.1724198Z Ran 1 test in 0.001s
2022-05-18T04:18:56.1724411Z 
2022-05-18T04:18:56.1724536Z OK (skipped=1)
2022-05-18T04:18:56.1724720Z 
2022-05-18T04:18:56.1724866Z Generating XML reports...
2022-05-18T04:18:56.1753498Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20220518041856.xml
2022-05-18T04:18:56.5167454Z Running distributed/test_data_parallel ... [2022-05-18 04:18:56.516356]
2022-05-18T04:18:56.5168018Z Executing ['/opt/conda/bin/python', 'distributed/test_data_parallel.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-05-18 04:18:56.516436]
2022-05-18T04:18:57.3480078Z Test results will be stored in test-reports/python-unittest/distributed.test_data_parallel
2022-05-18T04:18:57.3494025Z 
2022-05-18T04:18:57.3494127Z Running tests...
2022-05-18T04:18:57.3495196Z ----------------------------------------------------------------------
2022-05-18T04:18:57.3504087Z   test_autocast (__main__.TestDataParallel) ... skip: multi-GPU not supported (0.001s)
2022-05-18T04:18:57.3516026Z   test_data_parallel (__main__.TestDataParallel) ... skip: multi-GPU not supported (0.001s)
2022-05-18T04:18:57.3525480Z   test_data_parallel_buffers_requiring_grad (__main__.TestDataParallel) ... skip: multi-GPU not supported (0.001s)
2022-05-18T04:18:57.3534441Z   test_data_parallel_complex (__main__.TestDataParallel) ... skip: At least 2 CUDA GPUS needed (0.001s)
2022-05-18T04:18:57.3544277Z   test_data_parallel_device_args (__main__.TestDataParallel) ... skip: multi-GPU not supported (0.001s)
2022-05-18T04:18:57.3555673Z   test_data_parallel_function_deletion (__main__.TestDataParallel) ... skip: multi-GPU not supported (0.001s)
2022-05-18T04:18:57.3560610Z   test_data_parallel_lazy_linear (__main__.TestDataParallel) ... skip: multi-GPU not supported (0.000s)
2022-05-18T04:18:57.3590699Z   test_data_parallel_model_device (__main__.TestDataParallel)
2022-05-18T04:18:57.3591425Z Test device[0] check at forward time. ... skip: multi-GPU not supported (0.003s)
2022-05-18T04:18:57.3599250Z   test_data_parallel_model_no_refcycles (__main__.TestDataParallel) ... skip: multi-GPU not supported (0.001s)
2022-05-18T04:18:57.3608692Z   test_data_parallel_module_zero_inputs (__main__.TestDataParallel) ... skip: multi-GPU not supported (0.001s)
2022-05-18T04:18:57.3632619Z   test_data_parallel_multiple_input (__main__.TestDataParallel) ... skip: multi-GPU not supported (0.002s)
2022-05-18T04:18:57.3640470Z   test_data_parallel_nested_input (__main__.TestDataParallel) ... skip: multi-GPU not supported (0.001s)
2022-05-18T04:18:57.3656156Z   test_data_parallel_nested_output (__main__.TestDataParallel) ... skip: multi-GPU not supported (0.002s)
2022-05-18T04:18:57.3663570Z   test_data_parallel_no_grad (__main__.TestDataParallel) ... skip: multi-GPU not supported (0.001s)
2022-05-18T04:18:57.3676672Z   test_data_parallel_rnn (__main__.TestDataParallel) ... skip: multi-GPU not supported (0.001s)
2022-05-18T04:18:57.3681967Z   test_data_parallel_small_back (__main__.TestDataParallel) ... skip: multi-GPU not supported (0.001s)
2022-05-18T04:18:57.3693537Z   test_data_parallel_sparse (__main__.TestDataParallel) ... skip: multi-GPU not supported (0.001s)
2022-05-18T04:18:57.3695767Z   test_gather_cpu (__main__.TestDataParallel) ... skip: multi-GPU not supported (0.000s)
2022-05-18T04:18:57.3701244Z   test_gather_different_len_dicts (__main__.TestDataParallel) ... skip: multi-GPU not supported (0.001s)
2022-05-18T04:18:57.3703768Z   test_gather_gpu (__main__.TestDataParallel) ... skip: multi-GPU not supported (0.000s)
2022-05-18T04:18:57.3712547Z   test_parallel_apply (__main__.TestDataParallel) ... skip: multi-GPU not supported (0.001s)
2022-05-18T04:18:57.3721503Z   test_parallel_apply_autocast (__main__.TestDataParallel) ... skip: multi-GPU not supported (0.001s)
2022-05-18T04:18:57.3726562Z   test_parallel_apply_passes_exception (__main__.TestDataParallel) ... skip: CUDA unavailable (0.000s)
2022-05-18T04:18:57.3740481Z   test_parameter_list_dict_replica (__main__.TestDataParallel) ... skip: multi-GPU not supported (0.001s)
2022-05-18T04:18:57.3747595Z   test_replicate (__main__.TestDataParallel) ... skip: multi-GPU not supported (0.001s)
2022-05-18T04:18:57.3754670Z   test_replicate_buffers (__main__.TestDataParallel) ... skip: multi-GPU not supported (0.001s)
2022-05-18T04:18:57.3760146Z   test_save_replica_module (__main__.TestDataParallel) ... skip: multi-GPU not supported (0.001s)
2022-05-18T04:18:57.3762985Z   test_scatter_cpu (__main__.TestDataParallel) ... skip: multi-GPU not supported (0.000s)
2022-05-18T04:18:57.3765817Z   test_scatter_gpu (__main__.TestDataParallel) ... skip: multi-GPU not supported (0.000s)
2022-05-18T04:18:57.3799729Z   test_strided_grad_layout (__main__.TestDataParallel) ... skip: multi-GPU not supported (0.003s)
2022-05-18T04:18:57.3806048Z   test_zero_grad (__main__.TestDataParallel) ... skip: multi-GPU not supported (0.001s)
2022-05-18T04:18:57.3822854Z   test_data_parallel_module_cpu_float16 (__main__.TestDataParallelDeviceTypeCPU) ... skip: Only runs on cuda (0.002s)
2022-05-18T04:18:57.3832543Z   test_data_parallel_module_cpu_float32 (__main__.TestDataParallelDeviceTypeCPU) ... skip: Only runs on cuda (0.001s)
2022-05-18T04:18:57.3842045Z   test_data_parallel_module_cpu_float64 (__main__.TestDataParallelDeviceTypeCPU) ... skip: Only runs on cuda (0.001s)
2022-05-18T04:18:57.3853576Z   test_data_parallel_module_kwargs_only_cpu_float16 (__main__.TestDataParallelDeviceTypeCPU) ... skip: Only runs on cuda (0.001s)
2022-05-18T04:18:57.3865281Z   test_data_parallel_module_kwargs_only_cpu_float32 (__main__.TestDataParallelDeviceTypeCPU) ... skip: Only runs on cuda (0.001s)
2022-05-18T04:18:57.3876421Z   test_data_parallel_module_kwargs_only_cpu_float64 (__main__.TestDataParallelDeviceTypeCPU) ... skip: Only runs on cuda (0.001s)
2022-05-18T04:18:57.3888442Z   test_data_parallel_module_kwargs_only_empty_dict_cpu_float16 (__main__.TestDataParallelDeviceTypeCPU) ... skip: Only runs on cuda (0.001s)
2022-05-18T04:18:57.3900319Z   test_data_parallel_module_kwargs_only_empty_dict_cpu_float32 (__main__.TestDataParallelDeviceTypeCPU) ... skip: Only runs on cuda (0.001s)
2022-05-18T04:18:57.3912122Z   test_data_parallel_module_kwargs_only_empty_dict_cpu_float64 (__main__.TestDataParallelDeviceTypeCPU) ... skip: Only runs on cuda (0.001s)
2022-05-18T04:18:57.3924014Z   test_data_parallel_module_kwargs_only_empty_list_cpu_float16 (__main__.TestDataParallelDeviceTypeCPU) ... skip: Only runs on cuda (0.001s)
2022-05-18T04:18:57.3935797Z   test_data_parallel_module_kwargs_only_empty_list_cpu_float32 (__main__.TestDataParallelDeviceTypeCPU) ... skip: Only runs on cuda (0.001s)
2022-05-18T04:18:57.3947920Z   test_data_parallel_module_kwargs_only_empty_list_cpu_float64 (__main__.TestDataParallelDeviceTypeCPU) ... skip: Only runs on cuda (0.001s)
2022-05-18T04:18:57.3959725Z   test_data_parallel_module_kwargs_only_empty_tuple_cpu_float16 (__main__.TestDataParallelDeviceTypeCPU) ... skip: Only runs on cuda (0.001s)
2022-05-18T04:18:57.3971696Z   test_data_parallel_module_kwargs_only_empty_tuple_cpu_float32 (__main__.TestDataParallelDeviceTypeCPU) ... skip: Only runs on cuda (0.001s)
2022-05-18T04:18:57.3983845Z   test_data_parallel_module_kwargs_only_empty_tuple_cpu_float64 (__main__.TestDataParallelDeviceTypeCPU) ... skip: Only runs on cuda (0.001s)
2022-05-18T04:18:57.3984296Z 
2022-05-18T04:18:57.3984724Z ----------------------------------------------------------------------
2022-05-18T04:18:57.3985115Z Ran 46 tests in 0.049s
2022-05-18T04:18:57.3985228Z 
2022-05-18T04:18:57.3985290Z OK (skipped=46)
2022-05-18T04:18:57.3985396Z 
2022-05-18T04:18:57.3985482Z Generating XML reports...
2022-05-18T04:18:57.4036812Z Generated XML report: test-reports/python-unittest/distributed.test_data_parallel/TEST-TestDataParallel-20220518041857.xml
2022-05-18T04:18:57.4052650Z Generated XML report: test-reports/python-unittest/distributed.test_data_parallel/TEST-TestDataParallelDeviceTypeCPU-20220518041857.xml
2022-05-18T04:18:57.5688176Z Running distributed/test_distributed_spawn ... [2022-05-18 04:18:57.568466]
2022-05-18T04:18:57.5720628Z MPI not available -- MPI backend tests will be skipped
2022-05-18T04:18:57.5725667Z Running distributed tests for the test backend with env init_method
2022-05-18T04:18:57.5728943Z Executing ['/opt/conda/bin/python', 'distributed/test_distributed_spawn.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-05-18 04:18:57.572711]
2022-05-18T04:18:58.3031428Z <unittest.suite.TestSuite tests=[]>
2022-05-18T04:18:58.3997491Z Running distributed tests for the test backend with file init_method
2022-05-18T04:18:58.3998533Z Executing ['/opt/conda/bin/python', 'distributed/test_distributed_spawn.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-05-18 04:18:58.399652]
2022-05-18T04:18:59.1296919Z <unittest.suite.TestSuite tests=[]>
2022-05-18T04:18:59.2256379Z Running distributed tests for the gloo backend with env init_method
2022-05-18T04:18:59.2257428Z Executing ['/opt/conda/bin/python', 'distributed/test_distributed_spawn.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-05-18 04:18:59.225530]
2022-05-18T04:18:59.9558615Z <unittest.suite.TestSuite tests=[]>
2022-05-18T04:18:59.9597114Z <unittest.suite.TestSuite tests=[<__main__.TestDistBackendWithSpawn testMethod=test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager>, <__main__.TestDistBackendWithSpawn testMethod=test_3_level_hierarchical_model_averager>, <__main__.TestDistBackendWithSpawn testMethod=test_Backend_enum_class>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallelCPU>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallelCPU_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_2D_Input>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Channels_Last>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_No_Affine>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_non_default_stream>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_requires_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_with_amp_and_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedSampler_padding>, <__main__.TestDistBackendWithSpawn testMethod=test_SyncBatchNorm_process_group>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_allreduce_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_allreduce_with_then_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_simple>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_with_empty>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_multigpu_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_object_default_pg>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_object_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_max_complex_unsupported>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_complex_unsupported_ops>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_multigpu_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_result_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_async>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda_async>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_average_parameters>, <__main__.TestDistBackendWithSpawn testMethod=test_backend_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_backend_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_global>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_group>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_gloo>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_gloo_tags>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_mixed_backend_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_no_rank_zero_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_op_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_op_list_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_ring_exchange_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_self_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_tensor_err>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_group>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_object_list>, <__main__.TestDistBackendWithSpawn testMethod=test_compute_bucket_assignment_by_size_sparse_error_with_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_compute_bucket_assignment_by_size_sparse_error_without_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_broadcast_buffer>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_broadcast_buffer_via_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_buffer_hook_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_buffer_hook_allreduce_return_future>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_build_debug_param_to_name_mapping>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_build_debug_param_to_name_mapping_requires_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_comm_hook_logging>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_control_flow_different_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_control_flow_same_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_create_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_device>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_forward_backward_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_grad_div_uneven_inputs>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_allreduce_process_group>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_post_localSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_powerSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_ignore_params_arg>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_inference>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_join_model_equivalence>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_logging_data_cpu>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_logging_data_gpu>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_model_diff_num_params_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_model_diff_shape_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_multiple_nested_unused_params_err_ignore_params>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_multiple_nested_unused_params_error>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_namedtuple>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_new_tensor_in_fwd>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_new_tensor_in_fwd_static_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_profiling_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_profiling_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_python_error_logged>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_returns_tensor_with_no_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_shared_grad_acc_unused_params>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_static_graph_nested_types>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_sync_bn_training_vs_eval>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_sync_module_states>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_input_exception>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_input_join_disable>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_inputs>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_inputs_stop_iteration_sync_bn>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_unused_params_rebuild_buckets_exception>, <__main__.TestDistBackendWithSpawn testMethod=test_destroy_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_destroy_group>, <__main__.TestDistBackendWithSpawn testMethod=test_detect_ddp_is_actually_static>, <__main__.TestDistBackendWithSpawn testMethod=test_different_graph_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_dump_DDP_relevant_env_vars>, <__main__.TestDistBackendWithSpawn testMethod=test_gather>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_checks>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_group>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_object>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_object_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_get_backend>, <__main__.TestDistBackendWithSpawn testMethod=test_get_future>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank_size_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank_size_group>, <__main__.TestDistBackendWithSpawn testMethod=test_invalid_static_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_irecv>, <__main__.TestDistBackendWithSpawn testMethod=test_isend>, <__main__.TestDistBackendWithSpawn testMethod=test_isend_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_isend_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_allreduce_hang>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_allreduce_hang_wait_all_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_failure_order>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo_rank_0_timeout>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_wait_all_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_allgather>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_broadcast>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_reduce>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_high_priority_stream>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration_input_rank_exceeds_world_size>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration_negative_input_rank>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_group_size_exceeds_world_size>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_overlap_not_allowed>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_world_size_not_divisible_by_group_size>, <__main__.TestDistBackendWithSpawn testMethod=test_output_unused_in_loss_dict_module>, <__main__.TestDistBackendWithSpawn testMethod=test_output_unused_in_loss_tuple_module>, <__main__.TestDistBackendWithSpawn testMethod=test_periodic_model_averager>, <__main__.TestDistBackendWithSpawn testMethod=test_periodic_model_averager_param_group>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity_with_hierarchical_sgd>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_cuda_twice>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_twice>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_checks>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_group>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_object_list>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_sparse_all_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_sparse_all_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_stateless_api_with_ddp>, <__main__.TestDistBackendWithSpawn testMethod=test_static_graph_api_cpu>, <__main__.TestDistBackendWithSpawn testMethod=test_sync_bn_logged>, <__main__.TestDistBackendWithSpawn testMethod=test_undefined_grad_parity_unused_parameters>, <__main__.TestDistBackendWithSpawn testMethod=test_verify_model_across_rank_with_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_verify_model_across_rank_without_logger>]>
2022-05-18T04:18:59.9623804Z test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9624194Z test_3_level_hierarchical_model_averager (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9624499Z test_Backend_enum_class (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9624812Z test_DistributedDataParallel (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9625148Z test_DistributedDataParallelCPU (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9625488Z test_DistributedDataParallelCPU_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9625852Z test_DistributedDataParallel_SyncBatchNorm (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9626219Z test_DistributedDataParallel_SyncBatchNorm_2D_Input (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9626605Z test_DistributedDataParallel_SyncBatchNorm_Channels_Last (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9627001Z test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9627421Z test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9627893Z test_DistributedDataParallel_SyncBatchNorm_No_Affine (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9628319Z test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9628708Z test_DistributedDataParallel_non_default_stream (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9629069Z test_DistributedDataParallel_requires_grad (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9629434Z test_DistributedDataParallel_with_amp_and_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9629768Z test_DistributedSampler_padding (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9630087Z test_SyncBatchNorm_process_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9630402Z test_accumulate_gradients_no_sync (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9630718Z test_accumulate_gradients_no_sync_allreduce_hook (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9631081Z test_accumulate_gradients_no_sync_allreduce_with_then_hook (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9631433Z test_accumulate_gradients_no_sync_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9631738Z test_all_gather (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9632017Z test_all_gather_coalesced_complex (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9632332Z test_all_gather_coalesced_full_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9632645Z test_all_gather_coalesced_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9632941Z test_all_gather_coalesced_simple (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9633254Z test_all_gather_coalesced_with_empty (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9633556Z test_all_gather_complex (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9633839Z test_all_gather_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9634119Z test_all_gather_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9634414Z test_all_gather_full_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9634705Z test_all_gather_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9634979Z test_all_gather_multigpu (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9635278Z test_all_gather_multigpu_complex (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9635591Z test_all_gather_object_default_pg (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9635885Z test_all_gather_object_subgroup (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9636206Z test_all_reduce_coalesced_full_group_max (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9636529Z test_all_reduce_coalesced_full_group_min (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9636860Z test_all_reduce_coalesced_full_group_product (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9637177Z test_all_reduce_coalesced_full_group_sum (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9637495Z test_all_reduce_coalesced_group_max (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9637810Z test_all_reduce_coalesced_group_min (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9638118Z test_all_reduce_coalesced_group_product (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9638440Z test_all_reduce_coalesced_group_sum (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9638746Z test_all_reduce_coalesced_max (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9639072Z test_all_reduce_coalesced_max_complex_unsupported (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9639385Z test_all_reduce_coalesced_min (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9639688Z test_all_reduce_coalesced_product (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9639991Z test_all_reduce_coalesced_sum (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9640292Z test_all_reduce_complex_unsupported_ops (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9640654Z test_all_reduce_full_group_max (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9640950Z test_all_reduce_full_group_min (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9641278Z test_all_reduce_full_group_product (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9641657Z test_all_reduce_full_group_sum (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9641957Z test_all_reduce_group_max (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9642248Z test_all_reduce_group_min (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9642529Z test_all_reduce_group_product (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9642825Z test_all_reduce_group_sum (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9643107Z test_all_reduce_max (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9643371Z test_all_reduce_min (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9643658Z test_all_reduce_multigpu (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9643961Z test_all_reduce_multigpu_complex (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9644248Z test_all_reduce_product (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9644541Z test_all_reduce_result_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9644823Z test_all_reduce_sum (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9645103Z test_all_reduce_sum_async (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9645379Z test_all_reduce_sum_complex (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9645670Z test_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9645962Z test_all_reduce_sum_cuda_async (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9646253Z test_all_reduce_sum_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9646539Z test_all_to_all (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9646813Z test_all_to_all_complex (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9647081Z test_all_to_all_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9647366Z test_all_to_all_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9647658Z test_all_to_all_full_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9647955Z test_all_to_all_full_group_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9648238Z test_all_to_all_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9648522Z test_all_to_all_group_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9648819Z test_all_to_all_single_equal_split (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9649125Z test_all_to_all_single_equal_split_complex (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9649450Z test_all_to_all_single_equal_split_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9649779Z test_all_to_all_single_equal_split_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9650117Z test_all_to_all_single_equal_split_full_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9650443Z test_all_to_all_single_equal_split_full_group_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9650774Z test_all_to_all_single_equal_split_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9651102Z test_all_to_all_single_equal_split_group_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9651416Z test_all_to_all_single_unequal_split (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9651737Z test_all_to_all_single_unequal_split_complex (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9652059Z test_all_to_all_single_unequal_split_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9652389Z test_all_to_all_single_unequal_split_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9652714Z test_all_to_all_single_unequal_split_full_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9653050Z test_all_to_all_single_unequal_split_full_group_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9653427Z test_all_to_all_single_unequal_split_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9653782Z test_all_to_all_single_unequal_split_group_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9654100Z test_average_parameters (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9654389Z test_backend_full_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9654655Z test_backend_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9654924Z test_barrier (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9655194Z test_barrier_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9655476Z test_barrier_full_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9655752Z test_barrier_full_group_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9656040Z test_barrier_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9656323Z test_barrier_group_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9656607Z test_barrier_timeout_full_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9656906Z test_barrier_timeout_global (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9657202Z test_barrier_timeout_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9657482Z test_batch_isend_irecv_gloo (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9657782Z test_batch_isend_irecv_gloo_tags (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9658095Z test_batch_isend_irecv_mixed_backend_err (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9658401Z test_batch_isend_irecv_nccl (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9658692Z test_batch_isend_irecv_no_rank_zero_nccl (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9658995Z test_batch_isend_irecv_op_err (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9659298Z test_batch_isend_irecv_op_list_err (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9659602Z test_batch_isend_irecv_ring_exchange_nccl (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9659921Z test_batch_isend_irecv_self_nccl (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9660228Z test_batch_isend_irecv_tensor_err (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9660514Z test_broadcast (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9660777Z test_broadcast_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9661062Z test_broadcast_full_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9661349Z test_broadcast_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9661624Z test_broadcast_multigpu (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9661914Z test_broadcast_object_list (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9662248Z test_compute_bucket_assignment_by_size_sparse_error_with_logger (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9662615Z test_compute_bucket_assignment_by_size_sparse_error_without_logger (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9663125Z test_ddp_broadcast_buffer (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9663434Z test_ddp_broadcast_buffer_via_hook (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9663746Z test_ddp_buffer_hook_allreduce (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9664054Z test_ddp_buffer_hook_allreduce_return_future (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9664387Z test_ddp_build_debug_param_to_name_mapping (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9664738Z test_ddp_build_debug_param_to_name_mapping_requires_grad (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9665050Z test_ddp_comm_hook_logging (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9665365Z test_ddp_control_flow_different_across_ranks (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9665692Z test_ddp_control_flow_same_across_ranks (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9665999Z test_ddp_create_graph (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9666333Z test_ddp_device (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9666624Z test_ddp_forward_backward_hook (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9666976Z test_ddp_grad_div_uneven_inputs (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9667274Z test_ddp_hook_parity_allreduce (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9667605Z test_ddp_hook_parity_allreduce_process_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9667933Z test_ddp_hook_parity_post_localSGD (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9668227Z test_ddp_hook_parity_powerSGD (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9668567Z test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9668939Z test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9669359Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9669813Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9670265Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9670711Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9671158Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9671598Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9672046Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9672500Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9672904Z test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9673258Z test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9673590Z test_ddp_ignore_params_arg (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9673876Z test_ddp_inference (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9674173Z test_ddp_join_model_equivalence (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9674462Z test_ddp_logging_data_cpu (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9674755Z test_ddp_logging_data_gpu (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9675067Z test_ddp_model_diff_num_params_across_ranks (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9675384Z test_ddp_model_diff_shape_across_ranks (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9675728Z test_ddp_multiple_nested_unused_params_err_ignore_params (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9676073Z test_ddp_multiple_nested_unused_params_error (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9676380Z test_ddp_namedtuple (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9676657Z test_ddp_new_tensor_in_fwd (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9676962Z test_ddp_new_tensor_in_fwd_static_graph (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9677283Z test_ddp_profiling_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9677590Z test_ddp_profiling_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9677897Z test_ddp_python_error_logged (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9678204Z test_ddp_returns_tensor_with_no_grad (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9678547Z test_ddp_shared_grad_acc_unused_params (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9678893Z test_ddp_static_graph_nested_types (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9679215Z test_ddp_sync_bn_training_vs_eval (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9679514Z test_ddp_sync_module_states (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9679801Z test_ddp_uneven_input_exception (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9680109Z test_ddp_uneven_input_join_disable (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9680409Z test_ddp_uneven_inputs (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9680709Z test_ddp_uneven_inputs_stop_iteration_sync_bn (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9681049Z test_ddp_unused_params_rebuild_buckets_exception (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9681368Z test_destroy_full_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9681718Z test_destroy_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9682003Z test_detect_ddp_is_actually_static (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9682316Z test_different_graph_across_ranks (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9682626Z test_dump_DDP_relevant_env_vars (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9682894Z test_gather (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9683164Z test_gather_checks (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9683440Z test_gather_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9683708Z test_gather_full_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9683985Z test_gather_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9684262Z test_gather_object (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9684533Z test_gather_object_subgroup (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9684820Z test_get_backend (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9685087Z test_get_future (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9685351Z test_get_rank (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9685619Z test_get_rank_size_full_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9685910Z test_get_rank_size_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9686197Z test_invalid_static_graph (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9686456Z test_irecv (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9686713Z test_isend (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9686993Z test_isend_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9687276Z test_isend_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9687583Z test_monitored_barrier_allreduce_hang (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9687920Z test_monitored_barrier_allreduce_hang_wait_all_ranks (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9688259Z test_monitored_barrier_failure_order (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9688554Z test_monitored_barrier_gloo (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9688863Z test_monitored_barrier_gloo_rank_0_timeout (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9689185Z test_monitored_barrier_gloo_subgroup (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9689486Z test_monitored_barrier_wait_all_ranks (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9689801Z test_nccl_backend_bool_allgather (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9690109Z test_nccl_backend_bool_allreduce (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9690419Z test_nccl_backend_bool_broadcast (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9690705Z test_nccl_backend_bool_reduce (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9691008Z test_nccl_high_priority_stream (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9691345Z test_new_subgroups (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9691628Z test_new_subgroups_by_enumeration (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9692003Z test_new_subgroups_by_enumeration_input_rank_exceeds_world_size (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9692370Z test_new_subgroups_by_enumeration_negative_input_rank (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9692705Z test_new_subgroups_group_size_exceeds_world_size (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9693044Z test_new_subgroups_overlap_not_allowed (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9693383Z test_new_subgroups_world_size_not_divisible_by_group_size (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9693720Z test_output_unused_in_loss_dict_module (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9694024Z test_output_unused_in_loss_tuple_module (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9694335Z test_periodic_model_averager (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9694655Z test_periodic_model_averager_param_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9694965Z test_post_localSGD_optimizer_parity (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9695297Z test_post_localSGD_optimizer_parity_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9695651Z test_post_localSGD_optimizer_parity_with_hierarchical_sgd (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9696032Z test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9696357Z test_reduce_full_group_max (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9696648Z test_reduce_full_group_min (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9696945Z test_reduce_full_group_product (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9697232Z test_reduce_full_group_sum (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9697520Z test_reduce_group_max (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9697796Z test_reduce_group_min (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9698084Z test_reduce_group_product (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9698355Z test_reduce_group_sum (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9698625Z test_reduce_max (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9698892Z test_reduce_min (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9699158Z test_reduce_multigpu (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9699435Z test_reduce_product (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9699708Z test_reduce_sum (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9699967Z test_reduce_sum_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9700251Z test_reduce_sum_cuda_twice (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9700536Z test_reduce_sum_twice (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9700808Z test_scatter (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9701065Z test_scatter_checks (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9701347Z test_scatter_complex (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9701623Z test_scatter_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9701892Z test_scatter_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9702179Z test_scatter_full_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9702459Z test_scatter_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9702730Z test_scatter_object_list (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9703151Z test_send_recv (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9703430Z test_send_recv_any_source (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9703729Z test_send_recv_any_source_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9704060Z test_send_recv_any_source_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9704438Z test_send_recv_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9704728Z test_send_recv_nccl (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9705049Z test_send_recv_nccl_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9705366Z test_send_recv_nccl_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9705670Z test_send_recv_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9705949Z test_send_recv_with_tag (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9706254Z test_send_recv_with_tag_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9706575Z test_send_recv_with_tag_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9706878Z test_sparse_all_reduce_sum (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9707164Z test_sparse_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9707465Z test_stateless_api_with_ddp (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9707760Z test_static_graph_api_cpu (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9708034Z test_sync_bn_logged (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9708343Z test_undefined_grad_parity_unused_parameters (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9708674Z test_verify_model_across_rank_with_logger (__main__.TestDistBackendWithSpawn)
2022-05-18T04:18:59.9708988Z test_verify_model_across_rank_without_logger (__main__.TestDistBackendWithSpawn)
2022-05-18T04:19:00.6940979Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:19:00.6951266Z 
2022-05-18T04:19:00.6951346Z Running tests...
2022-05-18T04:19:00.6952270Z ----------------------------------------------------------------------
2022-05-18T04:19:00.9755015Z   test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10749
2022-05-18T04:19:00.9776727Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10750
2022-05-18T04:19:00.9835312Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10751
2022-05-18T04:19:01.7883368Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:19:01.7883836Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:19:01.7884499Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:19:01.7885116Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:19:01.7885956Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:19:01.7886831Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:19:01.7990155Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:19:01.8897769Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:19:01.8898387Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:19:02.0850113Z skip: Need at least 2 CUDA devices (1.390s)
2022-05-18T04:19:02.0850433Z 
2022-05-18T04:19:02.0850818Z ----------------------------------------------------------------------
2022-05-18T04:19:02.0851071Z Ran 1 test in 1.390s
2022-05-18T04:19:02.0851198Z 
2022-05-18T04:19:02.0851260Z OK (skipped=1)
2022-05-18T04:19:02.0851370Z 
2022-05-18T04:19:02.0851457Z Generating XML reports...
2022-05-18T04:19:02.0881436Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041900.xml
2022-05-18T04:19:03.0258432Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:19:03.0268931Z 
2022-05-18T04:19:03.0269081Z Running tests...
2022-05-18T04:19:03.0269738Z ----------------------------------------------------------------------
2022-05-18T04:19:03.0305362Z   test_3_level_hierarchical_model_averager (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.003s)
2022-05-18T04:19:03.0305918Z 
2022-05-18T04:19:03.0306390Z ----------------------------------------------------------------------
2022-05-18T04:19:03.0306861Z Ran 1 test in 0.004s
2022-05-18T04:19:03.0307087Z 
2022-05-18T04:19:03.0307212Z OK (skipped=1)
2022-05-18T04:19:03.0307343Z 
2022-05-18T04:19:03.0307433Z Generating XML reports...
2022-05-18T04:19:03.0337994Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041903.xml
2022-05-18T04:19:03.8656996Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:19:03.8666559Z 
2022-05-18T04:19:03.8666696Z Running tests...
2022-05-18T04:19:03.8667314Z ----------------------------------------------------------------------
2022-05-18T04:19:04.1475812Z   test_Backend_enum_class (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10812
2022-05-18T04:19:04.1498043Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10813
2022-05-18T04:19:04.1521157Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10814
2022-05-18T04:19:04.9819206Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:19:04.9819951Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:19:04.9820889Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:19:04.9821549Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:19:04.9822254Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:19:04.9822785Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:19:04.9926242Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:19:05.0833960Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:19:05.0834334Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:19:05.2572334Z ok (1.390s)
2022-05-18T04:19:05.2572549Z 
2022-05-18T04:19:05.2572974Z ----------------------------------------------------------------------
2022-05-18T04:19:05.2573454Z Ran 1 test in 1.390s
2022-05-18T04:19:05.2573676Z 
2022-05-18T04:19:05.2573748Z OK
2022-05-18T04:19:05.2573841Z 
2022-05-18T04:19:05.2573922Z Generating XML reports...
2022-05-18T04:19:05.2610651Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041903.xml
2022-05-18T04:19:06.1941202Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:19:06.1951669Z 
2022-05-18T04:19:06.1951788Z Running tests...
2022-05-18T04:19:06.1952514Z ----------------------------------------------------------------------
2022-05-18T04:19:06.4706030Z   test_DistributedDataParallel (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77317 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure IN_CI is not set and you are not using the flag --import-disabled-tests. (0.275s)
2022-05-18T04:19:06.4706857Z 
2022-05-18T04:19:06.4707067Z ----------------------------------------------------------------------
2022-05-18T04:19:06.4707333Z Ran 1 test in 0.275s
2022-05-18T04:19:06.4707448Z 
2022-05-18T04:19:06.4707611Z OK (skipped=1)
2022-05-18T04:19:06.4707721Z 
2022-05-18T04:19:06.4707809Z Generating XML reports...
2022-05-18T04:19:06.4733909Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041906.xml
2022-05-18T04:19:07.3755649Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:19:07.3765874Z 
2022-05-18T04:19:07.3765996Z Running tests...
2022-05-18T04:19:07.3766578Z ----------------------------------------------------------------------
2022-05-18T04:19:07.6606303Z   test_DistributedDataParallelCPU (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10875
2022-05-18T04:19:07.6628949Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10876
2022-05-18T04:19:07.6652356Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10877
2022-05-18T04:19:08.4921350Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:19:08.4921881Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:19:08.4922249Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:19:08.4922902Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:19:08.4923492Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:19:08.4924074Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:19:08.5028817Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:19:08.5096467Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8sfl7hx7
2022-05-18T04:19:08.5098920Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8sfl7hx7/_remote_module_non_scriptable.py
2022-05-18T04:19:08.5935463Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:19:08.5935974Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:19:08.6004170Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkn699kyt
2022-05-18T04:19:08.6005870Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkn699kyt/_remote_module_non_scriptable.py
2022-05-18T04:19:08.6007401Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa61t8o63
2022-05-18T04:19:08.6009724Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa61t8o63/_remote_module_non_scriptable.py
2022-05-18T04:19:08.6162261Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:19:08.6163016Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:19:08.6163689Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:19:08.8704814Z ok (1.494s)
2022-05-18T04:19:08.8704994Z 
2022-05-18T04:19:08.8705303Z ----------------------------------------------------------------------
2022-05-18T04:19:08.8705626Z Ran 1 test in 1.494s
2022-05-18T04:19:08.8705743Z 
2022-05-18T04:19:08.8705806Z OK
2022-05-18T04:19:08.8705901Z 
2022-05-18T04:19:08.8705998Z Generating XML reports...
2022-05-18T04:19:08.8736457Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041907.xml
2022-05-18T04:19:09.7926468Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:19:09.7936006Z 
2022-05-18T04:19:09.7936144Z Running tests...
2022-05-18T04:19:09.7936868Z ----------------------------------------------------------------------
2022-05-18T04:19:10.0753624Z   test_DistributedDataParallelCPU_grad_is_view (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10928
2022-05-18T04:19:10.0775158Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10929
2022-05-18T04:19:10.0798465Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10930
2022-05-18T04:19:10.8910611Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:19:10.9011327Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:19:10.9011740Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:19:10.9012379Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:19:10.9013029Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:19:10.9013911Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:19:10.9021395Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:19:10.9022587Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:19:10.9023942Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:19:10.9090675Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl7pbc8cu
2022-05-18T04:19:10.9092512Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl7pbc8cu/_remote_module_non_scriptable.py
2022-05-18T04:19:10.9121880Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqmv82jsc
2022-05-18T04:19:10.9122503Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphlxwykpn
2022-05-18T04:19:10.9124104Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqmv82jsc/_remote_module_non_scriptable.py
2022-05-18T04:19:10.9124827Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphlxwykpn/_remote_module_non_scriptable.py
2022-05-18T04:19:10.9334622Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:19:10.9335288Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:19:10.9335661Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:19:11.1848252Z ok (1.391s)
2022-05-18T04:19:11.1848466Z 
2022-05-18T04:19:11.1848892Z ----------------------------------------------------------------------
2022-05-18T04:19:11.1849376Z Ran 1 test in 1.391s
2022-05-18T04:19:11.1849581Z 
2022-05-18T04:19:11.1849671Z OK
2022-05-18T04:19:11.1849785Z 
2022-05-18T04:19:11.1849885Z Generating XML reports...
2022-05-18T04:19:11.1881410Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041909.xml
2022-05-18T04:19:12.1108910Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:19:12.1118360Z 
2022-05-18T04:19:12.1118468Z Running tests...
2022-05-18T04:19:12.1119682Z ----------------------------------------------------------------------
2022-05-18T04:19:12.3963870Z   test_DistributedDataParallel_SyncBatchNorm (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10981
2022-05-18T04:19:12.3985264Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10982
2022-05-18T04:19:12.4007532Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10983
2022-05-18T04:19:13.1873072Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:19:13.1950617Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:19:13.1951048Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:19:13.1951647Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:19:13.1952178Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:19:13.1974004Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:19:13.2059628Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:19:13.2060413Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:19:13.2987196Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:19:13.5058059Z skip: CUDA is not available. (1.394s)
2022-05-18T04:19:13.5058357Z 
2022-05-18T04:19:13.5058895Z ----------------------------------------------------------------------
2022-05-18T04:19:13.5059178Z Ran 1 test in 1.394s
2022-05-18T04:19:13.5059295Z 
2022-05-18T04:19:13.5059371Z OK (skipped=1)
2022-05-18T04:19:13.5059481Z 
2022-05-18T04:19:13.5059570Z Generating XML reports...
2022-05-18T04:19:13.5090347Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041912.xml
2022-05-18T04:19:14.4363666Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:19:14.4372971Z 
2022-05-18T04:19:14.4373104Z Running tests...
2022-05-18T04:19:14.4373575Z ----------------------------------------------------------------------
2022-05-18T04:19:14.7196332Z   test_DistributedDataParallel_SyncBatchNorm_2D_Input (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11034
2022-05-18T04:19:14.7218333Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11035
2022-05-18T04:19:14.7240735Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11036
2022-05-18T04:19:15.5219873Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:19:15.5321760Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:19:15.5322380Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:19:15.5323377Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:19:15.5324233Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:19:15.5324975Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:19:15.5334041Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:19:15.5334575Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:19:15.5335020Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:19:15.7290656Z skip: CUDA is not available. (1.291s)
2022-05-18T04:19:15.7291023Z 
2022-05-18T04:19:15.7291545Z ----------------------------------------------------------------------
2022-05-18T04:19:15.7291917Z Ran 1 test in 1.292s
2022-05-18T04:19:15.7292035Z 
2022-05-18T04:19:15.7292108Z OK (skipped=1)
2022-05-18T04:19:15.7292403Z 
2022-05-18T04:19:15.7292492Z Generating XML reports...
2022-05-18T04:19:15.7322735Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041914.xml
2022-05-18T04:19:16.6707269Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:19:16.6717564Z 
2022-05-18T04:19:16.6717900Z Running tests...
2022-05-18T04:19:16.6718300Z ----------------------------------------------------------------------
2022-05-18T04:19:16.9558423Z   test_DistributedDataParallel_SyncBatchNorm_Channels_Last (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11087
2022-05-18T04:19:16.9580732Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11088
2022-05-18T04:19:16.9604368Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11089
2022-05-18T04:19:17.8329288Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:19:17.8430282Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:19:17.8431171Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:19:17.8431620Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:19:17.8432122Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:19:17.8432639Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:19:17.8538191Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:19:17.9447115Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:19:17.9448094Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:19:18.1656527Z skip: CUDA is not available. (1.493s)
2022-05-18T04:19:18.1656833Z 
2022-05-18T04:19:18.1657354Z ----------------------------------------------------------------------
2022-05-18T04:19:18.1657605Z Ran 1 test in 1.494s
2022-05-18T04:19:18.1657720Z 
2022-05-18T04:19:18.1657794Z OK (skipped=1)
2022-05-18T04:19:18.1657904Z 
2022-05-18T04:19:18.1657989Z Generating XML reports...
2022-05-18T04:19:18.1688751Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041916.xml
2022-05-18T04:19:19.0857523Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:19:19.0868440Z 
2022-05-18T04:19:19.0868790Z Running tests...
2022-05-18T04:19:19.0869444Z ----------------------------------------------------------------------
2022-05-18T04:19:19.3682852Z   test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11140
2022-05-18T04:19:19.3704292Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11141
2022-05-18T04:19:19.3726782Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11142
2022-05-18T04:19:20.1967134Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:19:20.2068677Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:19:20.2069094Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:19:20.2069720Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:19:20.2070262Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:19:20.2071027Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:19:20.2078304Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:19:20.2078826Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:19:20.2080006Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:19:20.3774623Z skip: CUDA is not available. (1.290s)
2022-05-18T04:19:20.3774959Z 
2022-05-18T04:19:20.3775353Z ----------------------------------------------------------------------
2022-05-18T04:19:20.3775602Z Ran 1 test in 1.291s
2022-05-18T04:19:20.3775715Z 
2022-05-18T04:19:20.3775789Z OK (skipped=1)
2022-05-18T04:19:20.3775886Z 
2022-05-18T04:19:20.3775972Z Generating XML reports...
2022-05-18T04:19:20.3806752Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041919.xml
2022-05-18T04:19:21.3002522Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:19:21.3012284Z 
2022-05-18T04:19:21.3012803Z Running tests...
2022-05-18T04:19:21.3013201Z ----------------------------------------------------------------------
2022-05-18T04:19:21.5818409Z   test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11193
2022-05-18T04:19:21.5840216Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11194
2022-05-18T04:19:21.5863230Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11195
2022-05-18T04:19:22.3723645Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:19:22.3735631Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:19:22.3736205Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:19:22.3736835Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:19:22.3737355Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:19:22.3825150Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:19:22.3844120Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:19:22.3845180Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:19:22.4836782Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:19:22.6912880Z skip: CUDA is not available. (1.390s)
2022-05-18T04:19:22.6913142Z 
2022-05-18T04:19:22.6913533Z ----------------------------------------------------------------------
2022-05-18T04:19:22.6913997Z Ran 1 test in 1.390s
2022-05-18T04:19:22.6914214Z 
2022-05-18T04:19:22.6914316Z OK (skipped=1)
2022-05-18T04:19:22.6914424Z 
2022-05-18T04:19:22.6914497Z Generating XML reports...
2022-05-18T04:19:22.6945490Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041921.xml
2022-05-18T04:19:23.6147602Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:19:23.6157649Z 
2022-05-18T04:19:23.6157776Z Running tests...
2022-05-18T04:19:23.6158287Z ----------------------------------------------------------------------
2022-05-18T04:19:23.8965506Z   test_DistributedDataParallel_SyncBatchNorm_No_Affine (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11246
2022-05-18T04:19:23.8986939Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11247
2022-05-18T04:19:23.9009916Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11248
2022-05-18T04:19:24.7266585Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:19:24.7352106Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:19:24.7352601Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:19:24.7353231Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:19:24.7353748Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:19:24.7367833Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:19:24.7461982Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:19:24.7462691Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:19:24.8381833Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:19:25.0062391Z skip: CUDA is not available. (1.390s)
2022-05-18T04:19:25.0062720Z 
2022-05-18T04:19:25.0063311Z ----------------------------------------------------------------------
2022-05-18T04:19:25.0063567Z Ran 1 test in 1.390s
2022-05-18T04:19:25.0063671Z 
2022-05-18T04:19:25.0063748Z OK (skipped=1)
2022-05-18T04:19:25.0063855Z 
2022-05-18T04:19:25.0063941Z Generating XML reports...
2022-05-18T04:19:25.0094381Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041923.xml
2022-05-18T04:19:25.9342595Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:19:25.9353477Z 
2022-05-18T04:19:25.9353898Z Running tests...
2022-05-18T04:19:25.9354306Z ----------------------------------------------------------------------
2022-05-18T04:19:26.2166291Z   test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11299
2022-05-18T04:19:26.2188615Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11300
2022-05-18T04:19:26.2211808Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11301
2022-05-18T04:19:27.0472713Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:19:27.0571171Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:19:27.0571558Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:19:27.0572202Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:19:27.0572883Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:19:27.0573719Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:19:27.0680657Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:19:27.0681320Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:19:27.1587337Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:19:27.3263370Z skip: CUDA is not available. (1.391s)
2022-05-18T04:19:27.3263727Z 
2022-05-18T04:19:27.3264359Z ----------------------------------------------------------------------
2022-05-18T04:19:27.3264629Z Ran 1 test in 1.391s
2022-05-18T04:19:27.3264745Z 
2022-05-18T04:19:27.3264819Z OK (skipped=1)
2022-05-18T04:19:27.3264985Z 
2022-05-18T04:19:27.3265073Z Generating XML reports...
2022-05-18T04:19:27.3295371Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041925.xml
2022-05-18T04:19:28.2491076Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:19:28.2500932Z 
2022-05-18T04:19:28.2501299Z Running tests...
2022-05-18T04:19:28.2501710Z ----------------------------------------------------------------------
2022-05-18T04:19:28.5250975Z   test_DistributedDataParallel_non_default_stream (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/76428 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure IN_CI is not set and you are not using the flag --import-disabled-tests. (0.275s)
2022-05-18T04:19:28.5251652Z 
2022-05-18T04:19:28.5251916Z ----------------------------------------------------------------------
2022-05-18T04:19:28.5252243Z Ran 1 test in 0.275s
2022-05-18T04:19:28.5252361Z 
2022-05-18T04:19:28.5252421Z OK (skipped=1)
2022-05-18T04:19:28.5252527Z 
2022-05-18T04:19:28.5252623Z Generating XML reports...
2022-05-18T04:19:28.5279583Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041928.xml
2022-05-18T04:19:29.4194541Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:19:29.4204341Z 
2022-05-18T04:19:29.4204449Z Running tests...
2022-05-18T04:19:29.4205048Z ----------------------------------------------------------------------
2022-05-18T04:19:29.7005758Z   test_DistributedDataParallel_requires_grad (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11362
2022-05-18T04:19:29.7028782Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11363
2022-05-18T04:19:29.7051924Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11364
2022-05-18T04:19:30.4943585Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:19:30.5045313Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:19:30.5046023Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:19:30.5046889Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:19:30.5047427Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:19:30.5047962Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:19:30.5054999Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:19:30.5055659Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:19:30.5057754Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:19:30.8102784Z ok (1.389s)
2022-05-18T04:19:30.8103173Z 
2022-05-18T04:19:30.8103696Z ----------------------------------------------------------------------
2022-05-18T04:19:30.8104144Z Ran 1 test in 1.390s
2022-05-18T04:19:30.8104356Z 
2022-05-18T04:19:30.8104449Z OK
2022-05-18T04:19:30.8104616Z 
2022-05-18T04:19:30.8104786Z Generating XML reports...
2022-05-18T04:19:30.8136632Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041929.xml
2022-05-18T04:19:31.7322997Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:19:31.7333021Z 
2022-05-18T04:19:31.7333168Z Running tests...
2022-05-18T04:19:31.7334073Z ----------------------------------------------------------------------
2022-05-18T04:19:32.0076245Z   test_DistributedDataParallel_with_amp_and_grad_is_view (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77294 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure IN_CI is not set and you are not using the flag --import-disabled-tests. (0.274s)
2022-05-18T04:19:32.0076782Z 
2022-05-18T04:19:32.0077005Z ----------------------------------------------------------------------
2022-05-18T04:19:32.0077235Z Ran 1 test in 0.274s
2022-05-18T04:19:32.0077350Z 
2022-05-18T04:19:32.0077422Z OK (skipped=1)
2022-05-18T04:19:32.0077543Z 
2022-05-18T04:19:32.0077630Z Generating XML reports...
2022-05-18T04:19:32.0106045Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041931.xml
2022-05-18T04:19:32.9099764Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:19:32.9109774Z 
2022-05-18T04:19:32.9109901Z Running tests...
2022-05-18T04:19:32.9110484Z ----------------------------------------------------------------------
2022-05-18T04:19:33.1925825Z   test_DistributedSampler_padding (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11425
2022-05-18T04:19:33.1947839Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11426
2022-05-18T04:19:33.1970723Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11427
2022-05-18T04:19:34.0029860Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:19:34.0030651Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:19:34.0031310Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:19:34.0031719Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:19:34.0032218Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:19:34.0032732Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:19:34.0139449Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:19:34.0139851Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:19:34.1044312Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:19:34.3021478Z skip: Need at least 3 CUDA devices (1.391s)
2022-05-18T04:19:34.3021696Z 
2022-05-18T04:19:34.3022017Z ----------------------------------------------------------------------
2022-05-18T04:19:34.3022272Z Ran 1 test in 1.391s
2022-05-18T04:19:34.3022388Z 
2022-05-18T04:19:34.3022461Z OK (skipped=1)
2022-05-18T04:19:34.3022555Z 
2022-05-18T04:19:34.3022644Z Generating XML reports...
2022-05-18T04:19:34.3052964Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041932.xml
2022-05-18T04:19:35.2240550Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:19:35.2251020Z 
2022-05-18T04:19:35.2251495Z Running tests...
2022-05-18T04:19:35.2251906Z ----------------------------------------------------------------------
2022-05-18T04:19:35.2268311Z   test_SyncBatchNorm_process_group (__main__.TestDistBackendWithSpawn) ... skip: no torchvision (0.002s)
2022-05-18T04:19:35.2269068Z 
2022-05-18T04:19:35.2269438Z ----------------------------------------------------------------------
2022-05-18T04:19:35.2269692Z Ran 1 test in 0.002s
2022-05-18T04:19:35.2269891Z 
2022-05-18T04:19:35.2269966Z OK (skipped=1)
2022-05-18T04:19:35.2270062Z 
2022-05-18T04:19:35.2270147Z Generating XML reports...
2022-05-18T04:19:35.2300208Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041935.xml
2022-05-18T04:19:36.0600361Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:19:36.0610305Z 
2022-05-18T04:19:36.0610413Z Running tests...
2022-05-18T04:19:36.0610889Z ----------------------------------------------------------------------
2022-05-18T04:19:36.0625244Z   test_accumulate_gradients_no_sync (__main__.TestDistBackendWithSpawn)
2022-05-18T04:19:36.3410105Z Runs _test_accumulate_gradients_no_sync using default inputs ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11488
2022-05-18T04:19:36.3431593Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11489
2022-05-18T04:19:36.3454273Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11490
2022-05-18T04:19:37.1300757Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:19:37.1383583Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:19:37.1383979Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:19:37.1384676Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:19:37.1385214Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:19:37.1401396Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:19:37.1492211Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:19:37.1493165Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:19:37.1562941Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyj9_qryr
2022-05-18T04:19:37.1563969Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpllluk6jz
2022-05-18T04:19:37.1565126Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyj9_qryr/_remote_module_non_scriptable.py
2022-05-18T04:19:37.1566556Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpllluk6jz/_remote_module_non_scriptable.py
2022-05-18T04:19:37.2413119Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:19:37.2488104Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvumv646_
2022-05-18T04:19:37.2489524Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvumv646_/_remote_module_non_scriptable.py
2022-05-18T04:19:37.4505056Z ok (1.389s)
2022-05-18T04:19:37.4505266Z 
2022-05-18T04:19:37.4505680Z ----------------------------------------------------------------------
2022-05-18T04:19:37.4505934Z Ran 1 test in 1.389s
2022-05-18T04:19:37.4506053Z 
2022-05-18T04:19:37.4506106Z OK
2022-05-18T04:19:37.4506198Z 
2022-05-18T04:19:37.4506288Z Generating XML reports...
2022-05-18T04:19:37.4539170Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041936.xml
2022-05-18T04:19:38.3706391Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:19:38.3716254Z 
2022-05-18T04:19:38.3716374Z Running tests...
2022-05-18T04:19:38.3716936Z ----------------------------------------------------------------------
2022-05-18T04:19:38.3734680Z   test_accumulate_gradients_no_sync_allreduce_hook (__main__.TestDistBackendWithSpawn)
2022-05-18T04:19:38.6510509Z Runs multiple iterations on _test_accumulate_gradients_no_sync ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11541
2022-05-18T04:19:38.6532136Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11542
2022-05-18T04:19:38.6555551Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11543
2022-05-18T04:19:39.4753300Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:19:39.4854763Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:19:39.4855189Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:19:39.4855803Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:19:39.4856446Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:19:39.4857294Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:19:39.4865229Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:19:39.4865966Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:19:39.4867789Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:19:39.4937717Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe9v9o4h1
2022-05-18T04:19:39.4939584Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe9v9o4h1/_remote_module_non_scriptable.py
2022-05-18T04:19:39.4971257Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc4js3rzl
2022-05-18T04:19:39.4973508Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc4js3rzl/_remote_module_non_scriptable.py
2022-05-18T04:19:39.4985080Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdabp7j6i
2022-05-18T04:19:39.4987346Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdabp7j6i/_remote_module_non_scriptable.py
2022-05-18T04:19:39.5202018Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:19:39.5202744Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:19:39.5203436Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:19:39.6603354Z ok (1.288s)
2022-05-18T04:19:39.6603506Z 
2022-05-18T04:19:39.6603857Z ----------------------------------------------------------------------
2022-05-18T04:19:39.6604112Z Ran 1 test in 1.289s
2022-05-18T04:19:39.6604255Z 
2022-05-18T04:19:39.6604318Z OK
2022-05-18T04:19:39.6604411Z 
2022-05-18T04:19:39.6604495Z Generating XML reports...
2022-05-18T04:19:39.6635029Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041938.xml
2022-05-18T04:19:40.5842792Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:19:40.5852996Z 
2022-05-18T04:19:40.5853170Z Running tests...
2022-05-18T04:19:40.5853498Z ----------------------------------------------------------------------
2022-05-18T04:19:40.5872253Z   test_accumulate_gradients_no_sync_allreduce_with_then_hook (__main__.TestDistBackendWithSpawn)
2022-05-18T04:19:40.8640042Z Runs multiple iterations on _test_accumulate_gradients_no_sync using allreduce ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11594
2022-05-18T04:19:40.8661208Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11595
2022-05-18T04:19:40.8684072Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11596
2022-05-18T04:19:41.6685931Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:19:41.6787291Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:19:41.6787831Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:19:41.6788797Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:19:41.6789332Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:19:41.6789839Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:19:41.6797314Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:19:41.6798795Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:19:41.6799645Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:19:41.6868076Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjw18miju
2022-05-18T04:19:41.6869974Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjw18miju/_remote_module_non_scriptable.py
2022-05-18T04:19:41.6915707Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8cvjvl0a
2022-05-18T04:19:41.6916371Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnja3402s
2022-05-18T04:19:41.6917464Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8cvjvl0a/_remote_module_non_scriptable.py
2022-05-18T04:19:41.6918135Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnja3402s/_remote_module_non_scriptable.py
2022-05-18T04:19:41.7133523Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:19:41.7134123Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:19:41.7134485Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:19:41.9734790Z ok (1.388s)
2022-05-18T04:19:41.9735016Z 
2022-05-18T04:19:41.9735509Z ----------------------------------------------------------------------
2022-05-18T04:19:41.9735967Z Ran 1 test in 1.388s
2022-05-18T04:19:41.9736143Z 
2022-05-18T04:19:41.9736210Z OK
2022-05-18T04:19:41.9736302Z 
2022-05-18T04:19:41.9736386Z Generating XML reports...
2022-05-18T04:19:41.9766796Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041940.xml
2022-05-18T04:19:42.9031363Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:19:42.9041939Z 
2022-05-18T04:19:42.9042074Z Running tests...
2022-05-18T04:19:42.9042633Z ----------------------------------------------------------------------
2022-05-18T04:19:42.9057401Z   test_accumulate_gradients_no_sync_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-05-18T04:19:43.1848938Z Runs _test_accumulate_gradients_no_sync using default inputs ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11647
2022-05-18T04:19:43.1871303Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11648
2022-05-18T04:19:43.1894273Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11649
2022-05-18T04:19:43.9638594Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:19:43.9738214Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:19:43.9738908Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:19:43.9739767Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:19:43.9740397Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:19:43.9740933Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:19:43.9846088Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:19:43.9931425Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpntfdzml2
2022-05-18T04:19:43.9933680Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpntfdzml2/_remote_module_non_scriptable.py
2022-05-18T04:19:44.0752085Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:19:44.0752491Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:19:44.0832447Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpil5cba9k
2022-05-18T04:19:44.0832956Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqdpmjsqj
2022-05-18T04:19:44.0834006Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpil5cba9k/_remote_module_non_scriptable.py
2022-05-18T04:19:44.0834468Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqdpmjsqj/_remote_module_non_scriptable.py
2022-05-18T04:19:44.2945341Z ok (1.390s)
2022-05-18T04:19:44.2945568Z 
2022-05-18T04:19:44.2945901Z ----------------------------------------------------------------------
2022-05-18T04:19:44.2946156Z Ran 1 test in 1.390s
2022-05-18T04:19:44.2946260Z 
2022-05-18T04:19:44.2946321Z OK
2022-05-18T04:19:44.2946414Z 
2022-05-18T04:19:44.2946509Z Generating XML reports...
2022-05-18T04:19:44.2977323Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041942.xml
2022-05-18T04:19:45.2159205Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:19:45.2170166Z 
2022-05-18T04:19:45.2170442Z Running tests...
2022-05-18T04:19:45.2171101Z ----------------------------------------------------------------------
2022-05-18T04:19:45.5021492Z   test_all_gather (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11700
2022-05-18T04:19:45.5042946Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11701
2022-05-18T04:19:45.5066578Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11702
2022-05-18T04:19:46.3163670Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:19:46.3265060Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:19:46.3265503Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:19:46.3266128Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:19:46.3266670Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:19:46.3267196Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:19:46.3274628Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:19:46.3276392Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:19:46.3277170Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:19:46.6118621Z ok (1.395s)
2022-05-18T04:19:46.6118890Z 
2022-05-18T04:19:46.6119618Z ----------------------------------------------------------------------
2022-05-18T04:19:46.6119859Z Ran 1 test in 1.395s
2022-05-18T04:19:46.6119978Z 
2022-05-18T04:19:46.6120039Z OK
2022-05-18T04:19:46.6120204Z 
2022-05-18T04:19:46.6120301Z Generating XML reports...
2022-05-18T04:19:46.6149912Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041945.xml
2022-05-18T04:19:47.5313977Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:19:47.5324228Z 
2022-05-18T04:19:47.5324383Z Running tests...
2022-05-18T04:19:47.5324740Z ----------------------------------------------------------------------
2022-05-18T04:19:47.8163988Z   test_all_gather_coalesced_complex (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11756
2022-05-18T04:19:47.8186275Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11757
2022-05-18T04:19:47.8210537Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11758
2022-05-18T04:19:48.6263702Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:19:48.6364919Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:19:48.6365566Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:19:48.6366529Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:19:48.6367203Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:19:48.6367726Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:19:48.6375630Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:19:48.6376341Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:19:48.6376921Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:19:48.9261667Z ok (1.393s)
2022-05-18T04:19:48.9261881Z 
2022-05-18T04:19:48.9262501Z ----------------------------------------------------------------------
2022-05-18T04:19:48.9263044Z Ran 1 test in 1.394s
2022-05-18T04:19:48.9263229Z 
2022-05-18T04:19:48.9263331Z OK
2022-05-18T04:19:48.9263471Z 
2022-05-18T04:19:48.9263604Z Generating XML reports...
2022-05-18T04:19:48.9294850Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041947.xml
2022-05-18T04:19:49.8521155Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:19:49.8531833Z 
2022-05-18T04:19:49.8532154Z Running tests...
2022-05-18T04:19:50.1372117Z ----------------------------------------------------------------------
2022-05-18T04:19:50.1372633Z   test_all_gather_coalesced_full_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11812
2022-05-18T04:19:50.1394970Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11813
2022-05-18T04:19:50.1417388Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11814
2022-05-18T04:19:50.9234135Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:19:50.9335494Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:19:50.9336138Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:19:50.9337118Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:19:50.9338105Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:19:50.9338732Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:19:50.9347373Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:19:50.9348012Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:19:50.9348597Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:19:50.9552893Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:19:50.9654679Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-05-18T04:19:50.9655358Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:19:50.9656115Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:19:50.9656639Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:19:50.9657161Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:19:51.2466237Z ok (1.393s)
2022-05-18T04:19:51.2466496Z 
2022-05-18T04:19:51.2466910Z ----------------------------------------------------------------------
2022-05-18T04:19:51.2467171Z Ran 1 test in 1.393s
2022-05-18T04:19:51.2467290Z 
2022-05-18T04:19:51.2467357Z OK
2022-05-18T04:19:51.2467437Z 
2022-05-18T04:19:51.2467532Z Generating XML reports...
2022-05-18T04:19:51.2500657Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041949.xml
2022-05-18T04:19:52.1665420Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:19:52.1676538Z 
2022-05-18T04:19:52.1676840Z Running tests...
2022-05-18T04:19:52.1677510Z ----------------------------------------------------------------------
2022-05-18T04:19:52.4569000Z   test_all_gather_coalesced_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11877
2022-05-18T04:19:52.4591128Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11878
2022-05-18T04:19:52.4615391Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11879
2022-05-18T04:19:53.2537506Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:19:53.2626270Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:19:53.2626778Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:19:53.2627442Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:19:53.2627975Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:19:53.2638180Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:19:53.2736177Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:19:53.2736553Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:19:53.2941587Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:19:53.2942266Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-05-18T04:19:53.3649317Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:19:53.3650638Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:19:53.3651424Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:19:53.3750222Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:19:53.3750795Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:19:53.6667833Z ok (1.499s)
2022-05-18T04:19:53.6668110Z 
2022-05-18T04:19:53.6668657Z ----------------------------------------------------------------------
2022-05-18T04:19:53.6668913Z Ran 1 test in 1.499s
2022-05-18T04:19:53.6669029Z 
2022-05-18T04:19:53.6669092Z OK
2022-05-18T04:19:53.6669171Z 
2022-05-18T04:19:53.6669283Z Generating XML reports...
2022-05-18T04:19:53.6700351Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041952.xml
2022-05-18T04:19:54.5909794Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:19:54.5919698Z 
2022-05-18T04:19:54.5919826Z Running tests...
2022-05-18T04:19:54.5920402Z ----------------------------------------------------------------------
2022-05-18T04:19:54.8736003Z   test_all_gather_coalesced_simple (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11938
2022-05-18T04:19:54.8758993Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11939
2022-05-18T04:19:54.8781345Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11940
2022-05-18T04:19:55.6996724Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:19:55.7079077Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:19:55.7079803Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:19:55.7080604Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:19:55.7081186Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:19:55.7098142Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:19:55.7188160Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:19:55.7188805Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:19:55.8108621Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:19:56.0833853Z ok (1.491s)
2022-05-18T04:19:56.0834032Z 
2022-05-18T04:19:56.0834510Z ----------------------------------------------------------------------
2022-05-18T04:19:56.0834765Z Ran 1 test in 1.491s
2022-05-18T04:19:56.0834880Z 
2022-05-18T04:19:56.0834941Z OK
2022-05-18T04:19:56.0835076Z 
2022-05-18T04:19:56.0835184Z Generating XML reports...
2022-05-18T04:19:56.0865375Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041954.xml
2022-05-18T04:19:57.0012142Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:19:57.0022201Z 
2022-05-18T04:19:57.0022327Z Running tests...
2022-05-18T04:19:57.0022761Z ----------------------------------------------------------------------
2022-05-18T04:19:57.2865792Z   test_all_gather_coalesced_with_empty (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11994
2022-05-18T04:19:57.2888302Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11995
2022-05-18T04:19:57.2912244Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11996
2022-05-18T04:19:58.0672456Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:19:58.0773258Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:19:58.0774228Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:19:58.0774969Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:19:58.0775519Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:19:58.0776031Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:19:58.0783879Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:19:58.0784387Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:19:58.0785179Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:19:58.3964258Z ok (1.394s)
2022-05-18T04:19:58.3964490Z 
2022-05-18T04:19:58.3964991Z ----------------------------------------------------------------------
2022-05-18T04:19:58.3965345Z Ran 1 test in 1.394s
2022-05-18T04:19:58.3965463Z 
2022-05-18T04:19:58.3965523Z OK
2022-05-18T04:19:58.3965617Z 
2022-05-18T04:19:58.3965697Z Generating XML reports...
2022-05-18T04:19:58.3996280Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041956.xml
2022-05-18T04:19:59.3189011Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:19:59.3199867Z 
2022-05-18T04:19:59.3200166Z Running tests...
2022-05-18T04:19:59.3200817Z ----------------------------------------------------------------------
2022-05-18T04:19:59.6068663Z   test_all_gather_complex (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12050
2022-05-18T04:19:59.6090787Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12051
2022-05-18T04:19:59.6114602Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12052
2022-05-18T04:20:00.4115874Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:20:00.4216993Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:20:00.4217584Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:20:00.4218520Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:20:00.4219387Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:20:00.4220233Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:20:00.4325112Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:20:00.5231090Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:20:00.5231480Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:20:00.8168190Z ok (1.497s)
2022-05-18T04:20:00.8168389Z 
2022-05-18T04:20:00.8168872Z ----------------------------------------------------------------------
2022-05-18T04:20:00.8169273Z Ran 1 test in 1.497s
2022-05-18T04:20:00.8169707Z 
2022-05-18T04:20:00.8169808Z OK
2022-05-18T04:20:00.8169955Z 
2022-05-18T04:20:00.8170106Z Generating XML reports...
2022-05-18T04:20:00.8201900Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041959.xml
2022-05-18T04:20:01.7397598Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:20:01.7407506Z 
2022-05-18T04:20:01.7407587Z Running tests...
2022-05-18T04:20:01.7408160Z ----------------------------------------------------------------------
2022-05-18T04:20:01.7424092Z   test_all_gather_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all gather (0.001s)
2022-05-18T04:20:01.7424385Z 
2022-05-18T04:20:01.7424792Z ----------------------------------------------------------------------
2022-05-18T04:20:01.7425236Z Ran 1 test in 0.002s
2022-05-18T04:20:01.7425426Z 
2022-05-18T04:20:01.7425538Z OK (skipped=1)
2022-05-18T04:20:01.7425672Z 
2022-05-18T04:20:01.7425758Z Generating XML reports...
2022-05-18T04:20:01.7456224Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042001.xml
2022-05-18T04:20:02.5750089Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:20:02.5760320Z 
2022-05-18T04:20:02.5760444Z Running tests...
2022-05-18T04:20:02.5761089Z ----------------------------------------------------------------------
2022-05-18T04:20:02.5777532Z   test_all_gather_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all gather (0.001s)
2022-05-18T04:20:02.5777969Z 
2022-05-18T04:20:02.5778332Z ----------------------------------------------------------------------
2022-05-18T04:20:02.5778581Z Ran 1 test in 0.002s
2022-05-18T04:20:02.5778698Z 
2022-05-18T04:20:02.5778772Z OK (skipped=1)
2022-05-18T04:20:02.5778881Z 
2022-05-18T04:20:02.5778954Z Generating XML reports...
2022-05-18T04:20:02.5808736Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042002.xml
2022-05-18T04:20:03.4052229Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:20:03.4062127Z 
2022-05-18T04:20:03.4062221Z Running tests...
2022-05-18T04:20:03.4062773Z ----------------------------------------------------------------------
2022-05-18T04:20:03.6895397Z   test_all_gather_full_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12126
2022-05-18T04:20:03.6918315Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12127
2022-05-18T04:20:03.6941848Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12128
2022-05-18T04:20:04.4849945Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:20:04.4951122Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:20:04.4951556Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:20:04.4952199Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:20:04.4952729Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:20:04.4953251Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:20:04.5059893Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:20:04.5060450Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:20:04.5963533Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:20:04.6173275Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:20:04.6174231Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-05-18T04:20:04.6174657Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:20:04.6175266Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:20:04.6175797Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:20:04.6176323Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:20:04.8993381Z ok (1.493s)
2022-05-18T04:20:04.8993629Z 
2022-05-18T04:20:04.8994144Z ----------------------------------------------------------------------
2022-05-18T04:20:04.8994531Z Ran 1 test in 1.493s
2022-05-18T04:20:04.8994647Z 
2022-05-18T04:20:04.8994711Z OK
2022-05-18T04:20:04.8994804Z 
2022-05-18T04:20:04.8994900Z Generating XML reports...
2022-05-18T04:20:04.9025889Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042003.xml
2022-05-18T04:20:05.8258852Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:20:05.8269218Z 
2022-05-18T04:20:05.8269343Z Running tests...
2022-05-18T04:20:05.8269808Z ----------------------------------------------------------------------
2022-05-18T04:20:06.1092154Z   test_all_gather_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12191
2022-05-18T04:20:06.1115681Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12192
2022-05-18T04:20:06.1139148Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12193
2022-05-18T04:20:06.8965358Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:20:06.8966073Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:20:06.8966663Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:20:06.8967448Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:20:06.8967987Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:20:06.8968508Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:20:06.8977047Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:20:06.8977511Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:20:06.8977865Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:20:06.8978232Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:20:06.9082968Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:20:06.9083559Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-05-18T04:20:06.9084493Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:20:06.9085250Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:20:06.9182226Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:20:07.3194025Z ok (1.492s)
2022-05-18T04:20:07.3194259Z 
2022-05-18T04:20:07.3194811Z ----------------------------------------------------------------------
2022-05-18T04:20:07.3195223Z Ran 1 test in 1.492s
2022-05-18T04:20:07.3195528Z 
2022-05-18T04:20:07.3195598Z OK
2022-05-18T04:20:07.3195692Z 
2022-05-18T04:20:07.3195791Z Generating XML reports...
2022-05-18T04:20:07.3227995Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042005.xml
2022-05-18T04:20:08.2414305Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:20:08.2424437Z 
2022-05-18T04:20:08.2424545Z Running tests...
2022-05-18T04:20:08.2425676Z ----------------------------------------------------------------------
2022-05-18T04:20:08.2441081Z   test_all_gather_multigpu (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl backend supports allgather multigpu (0.002s)
2022-05-18T04:20:08.2441478Z 
2022-05-18T04:20:08.2441869Z ----------------------------------------------------------------------
2022-05-18T04:20:08.2442146Z Ran 1 test in 0.002s
2022-05-18T04:20:08.2442261Z 
2022-05-18T04:20:08.2442335Z OK (skipped=1)
2022-05-18T04:20:08.2442457Z 
2022-05-18T04:20:08.2442572Z Generating XML reports...
2022-05-18T04:20:08.2473726Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042008.xml
2022-05-18T04:20:09.0739279Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:20:09.0749039Z 
2022-05-18T04:20:09.0749160Z Running tests...
2022-05-18T04:20:09.0749736Z ----------------------------------------------------------------------
2022-05-18T04:20:09.0766501Z   test_all_gather_multigpu_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl backend supports allgather multigpu (0.002s)
2022-05-18T04:20:09.0766775Z 
2022-05-18T04:20:09.0767032Z ----------------------------------------------------------------------
2022-05-18T04:20:09.0767292Z Ran 1 test in 0.002s
2022-05-18T04:20:09.0767407Z 
2022-05-18T04:20:09.0767479Z OK (skipped=1)
2022-05-18T04:20:09.0767573Z 
2022-05-18T04:20:09.0767664Z Generating XML reports...
2022-05-18T04:20:09.0798614Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042009.xml
2022-05-18T04:20:09.9083525Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:20:09.9093944Z 
2022-05-18T04:20:09.9094403Z Running tests...
2022-05-18T04:20:09.9094790Z ----------------------------------------------------------------------
2022-05-18T04:20:10.1968800Z   test_all_gather_object_default_pg (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12272
2022-05-18T04:20:10.1990885Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12273
2022-05-18T04:20:10.2015217Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12274
2022-05-18T04:20:11.0211455Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:20:11.0313188Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:20:11.0313817Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:20:11.0314192Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:20:11.0314701Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:20:11.0315216Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:20:11.0322898Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:20:11.0323584Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:20:11.0324796Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:20:11.2063826Z ok (1.297s)
2022-05-18T04:20:11.2064347Z 
2022-05-18T04:20:11.2064864Z ----------------------------------------------------------------------
2022-05-18T04:20:11.2065317Z Ran 1 test in 1.297s
2022-05-18T04:20:11.2065437Z 
2022-05-18T04:20:11.2065675Z OK
2022-05-18T04:20:11.2065819Z 
2022-05-18T04:20:11.2065978Z Generating XML reports...
2022-05-18T04:20:11.2104924Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042009.xml
2022-05-18T04:20:12.1259267Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:20:12.1269216Z 
2022-05-18T04:20:12.1269344Z Running tests...
2022-05-18T04:20:12.1269929Z ----------------------------------------------------------------------
2022-05-18T04:20:12.4125156Z   test_all_gather_object_subgroup (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12325
2022-05-18T04:20:12.4147474Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12326
2022-05-18T04:20:12.4172131Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12327
2022-05-18T04:20:13.2073042Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:20:13.2167827Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:20:13.2168304Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:20:13.2168925Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:20:13.2169448Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:20:13.2174306Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:20:13.2277907Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:20:13.2278708Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:20:13.3186430Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:20:13.3698218Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:20:13.3799083Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:20:13.3799772Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-05-18T04:20:13.3800896Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:20:13.3801449Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:20:13.3801966Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:20:13.4144744Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-05-18T04:20:13.4246438Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 2
2022-05-18T04:20:13.4247183Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-05-18T04:20:13.4248270Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 3 nodes.
2022-05-18T04:20:13.4248857Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:3 with 3 nodes.
2022-05-18T04:20:13.4249896Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 3 nodes.
2022-05-18T04:20:13.4370141Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 0
2022-05-18T04:20:13.4471472Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 2
2022-05-18T04:20:13.4472083Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 1
2022-05-18T04:20:13.4472965Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:4 with 3 nodes.
2022-05-18T04:20:13.4473771Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:4 with 3 nodes.
2022-05-18T04:20:13.4474583Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:4 with 3 nodes.
2022-05-18T04:20:13.7226208Z ok (1.595s)
2022-05-18T04:20:13.7226421Z 
2022-05-18T04:20:13.7226854Z ----------------------------------------------------------------------
2022-05-18T04:20:13.7227141Z Ran 1 test in 1.596s
2022-05-18T04:20:13.7227264Z 
2022-05-18T04:20:13.7227326Z OK
2022-05-18T04:20:13.7227419Z 
2022-05-18T04:20:13.7227510Z Generating XML reports...
2022-05-18T04:20:13.7258444Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042012.xml
2022-05-18T04:20:14.6383144Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:20:14.6392017Z 
2022-05-18T04:20:14.6392146Z Running tests...
2022-05-18T04:20:14.6392707Z ----------------------------------------------------------------------
2022-05-18T04:20:14.9226241Z   test_all_reduce_coalesced_full_group_max (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12414
2022-05-18T04:20:14.9247999Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12415
2022-05-18T04:20:14.9270548Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12416
2022-05-18T04:20:15.7231696Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:20:15.7267122Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:20:15.7267657Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:20:15.7268358Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:20:15.7268893Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:20:15.7332634Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:20:15.7376707Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:20:15.7377125Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:20:15.8343302Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:20:15.8590597Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:20:15.8591205Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:20:15.8591859Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-05-18T04:20:15.8592755Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:20:15.8593766Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:20:15.8594361Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:20:16.1324489Z ok (1.493s)
2022-05-18T04:20:16.1324755Z 
2022-05-18T04:20:16.1325266Z ----------------------------------------------------------------------
2022-05-18T04:20:16.1325704Z Ran 1 test in 1.493s
2022-05-18T04:20:16.1325916Z 
2022-05-18T04:20:16.1326029Z OK
2022-05-18T04:20:16.1326198Z 
2022-05-18T04:20:16.1326372Z Generating XML reports...
2022-05-18T04:20:16.1356223Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042014.xml
2022-05-18T04:20:17.0629484Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:20:17.0639317Z 
2022-05-18T04:20:17.0639431Z Running tests...
2022-05-18T04:20:17.0640143Z ----------------------------------------------------------------------
2022-05-18T04:20:17.3501025Z   test_all_reduce_coalesced_full_group_min (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12479
2022-05-18T04:20:17.3523481Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12480
2022-05-18T04:20:17.3548245Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12481
2022-05-18T04:20:18.1607863Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:20:18.1608509Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:20:18.1608888Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:20:18.1609501Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:20:18.1610324Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:20:18.1611108Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:20:18.1618011Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:20:18.1618606Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:20:18.2619508Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:20:18.2926342Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:20:18.3028236Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:20:18.3028895Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-05-18T04:20:18.3029769Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:20:18.3030664Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:20:18.3031440Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:20:18.5601181Z ok (1.496s)
2022-05-18T04:20:18.5601398Z 
2022-05-18T04:20:18.5601818Z ----------------------------------------------------------------------
2022-05-18T04:20:18.5602262Z Ran 1 test in 1.496s
2022-05-18T04:20:18.5602438Z 
2022-05-18T04:20:18.5602504Z OK
2022-05-18T04:20:18.5602588Z 
2022-05-18T04:20:18.5602683Z Generating XML reports...
2022-05-18T04:20:18.5633549Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042017.xml
2022-05-18T04:20:19.4744955Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:20:19.4754350Z 
2022-05-18T04:20:19.4754589Z Running tests...
2022-05-18T04:20:19.4755243Z ----------------------------------------------------------------------
2022-05-18T04:20:19.7580886Z   test_all_reduce_coalesced_full_group_product (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12544
2022-05-18T04:20:19.7602995Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12545
2022-05-18T04:20:19.7627448Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12546
2022-05-18T04:20:20.5631573Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:20:20.5733002Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:20:20.5733603Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:20:20.5734285Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:20:20.5734860Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:20:20.5735386Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:20:20.5743704Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:20:20.5744678Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:20:20.5745591Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:20:20.5950947Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:20:20.6052991Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-05-18T04:20:20.6053433Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:20:20.6054035Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:20:20.6054566Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:20:20.6055090Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:20:20.8681535Z ok (1.392s)
2022-05-18T04:20:20.8681749Z 
2022-05-18T04:20:20.8682147Z ----------------------------------------------------------------------
2022-05-18T04:20:20.8682384Z Ran 1 test in 1.393s
2022-05-18T04:20:20.8682500Z 
2022-05-18T04:20:20.8682565Z OK
2022-05-18T04:20:20.8682672Z 
2022-05-18T04:20:20.8682772Z Generating XML reports...
2022-05-18T04:20:20.8714222Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042019.xml
2022-05-18T04:20:21.7932978Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:20:21.7942479Z 
2022-05-18T04:20:21.7943074Z Running tests...
2022-05-18T04:20:21.7943469Z ----------------------------------------------------------------------
2022-05-18T04:20:22.0739052Z   test_all_reduce_coalesced_full_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12609
2022-05-18T04:20:22.0761839Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12610
2022-05-18T04:20:22.0785473Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12611
2022-05-18T04:20:22.8770028Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:20:22.8871393Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:20:22.8872228Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:20:22.8873186Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:20:22.8873964Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:20:22.8874495Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:20:22.8978385Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:20:22.9884621Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:20:22.9885230Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:20:23.0092381Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:20:23.0092972Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:20:23.0093524Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-05-18T04:20:23.0094390Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:20:23.0095281Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:20:23.0096102Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:20:23.2840053Z ok (1.489s)
2022-05-18T04:20:23.2840327Z 
2022-05-18T04:20:23.2840910Z ----------------------------------------------------------------------
2022-05-18T04:20:23.2841169Z Ran 1 test in 1.490s
2022-05-18T04:20:23.2841286Z 
2022-05-18T04:20:23.2841334Z OK
2022-05-18T04:20:23.2841426Z 
2022-05-18T04:20:23.2841531Z Generating XML reports...
2022-05-18T04:20:23.2871525Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042021.xml
2022-05-18T04:20:24.2061734Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:20:24.2072020Z 
2022-05-18T04:20:24.2072122Z Running tests...
2022-05-18T04:20:24.2073065Z ----------------------------------------------------------------------
2022-05-18T04:20:24.4886841Z   test_all_reduce_coalesced_group_max (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12674
2022-05-18T04:20:24.4909495Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12675
2022-05-18T04:20:24.4933742Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12676
2022-05-18T04:20:25.2759137Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:20:25.2859135Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:20:25.2859602Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:20:25.2860229Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:20:25.2860747Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:20:25.2861274Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:20:25.2967908Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:20:25.3873888Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:20:25.3874699Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:20:25.3875663Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:20:25.4081251Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-05-18T04:20:25.4081905Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:20:25.4082634Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:20:25.4083158Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:20:25.4181280Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:20:25.6985883Z ok (1.491s)
2022-05-18T04:20:25.6986134Z 
2022-05-18T04:20:25.6986539Z ----------------------------------------------------------------------
2022-05-18T04:20:25.6986800Z Ran 1 test in 1.491s
2022-05-18T04:20:25.6986921Z 
2022-05-18T04:20:25.6986984Z OK
2022-05-18T04:20:25.6987065Z 
2022-05-18T04:20:25.6987163Z Generating XML reports...
2022-05-18T04:20:25.7017627Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042024.xml
2022-05-18T04:20:26.6279789Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:20:26.6289846Z 
2022-05-18T04:20:26.6289970Z Running tests...
2022-05-18T04:20:26.6290419Z ----------------------------------------------------------------------
2022-05-18T04:20:26.9129328Z   test_all_reduce_coalesced_group_min (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12735
2022-05-18T04:20:26.9151329Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12736
2022-05-18T04:20:26.9175203Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12737
2022-05-18T04:20:27.6921049Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:20:27.6921766Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:20:27.6922395Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:20:27.6923331Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:20:27.6923862Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:20:27.6924435Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:20:27.6931398Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:20:27.6932434Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:20:27.6933694Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:20:27.6934348Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:20:27.7040478Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-05-18T04:20:27.7040976Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:20:27.7041560Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:20:27.7042267Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:20:27.7137233Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:20:28.0226624Z ok (1.393s)
2022-05-18T04:20:28.0226896Z 
2022-05-18T04:20:28.0227405Z ----------------------------------------------------------------------
2022-05-18T04:20:28.0227695Z Ran 1 test in 1.394s
2022-05-18T04:20:28.0227814Z 
2022-05-18T04:20:28.0227877Z OK
2022-05-18T04:20:28.0227970Z 
2022-05-18T04:20:28.0229182Z Generating XML reports...
2022-05-18T04:20:28.0258195Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042026.xml
2022-05-18T04:20:28.9424371Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:20:28.9433843Z 
2022-05-18T04:20:28.9433966Z Running tests...
2022-05-18T04:20:28.9434846Z ----------------------------------------------------------------------
2022-05-18T04:20:29.2253296Z   test_all_reduce_coalesced_group_product (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12796
2022-05-18T04:20:29.2275791Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12797
2022-05-18T04:20:29.2298821Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12798
2022-05-18T04:20:30.0758700Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:20:30.0759384Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:20:30.0759996Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:20:30.0760993Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:20:30.0761639Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:20:30.0762209Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:20:30.0865365Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:20:30.1772916Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:20:30.1773427Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:20:30.1774194Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:20:30.1976534Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-05-18T04:20:30.1977212Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:20:30.1977984Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:20:30.1978605Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:20:30.1979133Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:20:30.5353804Z ok (1.592s)
2022-05-18T04:20:30.5354050Z 
2022-05-18T04:20:30.5354466Z ----------------------------------------------------------------------
2022-05-18T04:20:30.5354727Z Ran 1 test in 1.592s
2022-05-18T04:20:30.5354854Z 
2022-05-18T04:20:30.5354920Z OK
2022-05-18T04:20:30.5355000Z 
2022-05-18T04:20:30.5355091Z Generating XML reports...
2022-05-18T04:20:30.5385335Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042028.xml
2022-05-18T04:20:31.4585409Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:20:31.4595347Z 
2022-05-18T04:20:31.4595585Z Running tests...
2022-05-18T04:20:31.4596239Z ----------------------------------------------------------------------
2022-05-18T04:20:31.7440246Z   test_all_reduce_coalesced_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12857
2022-05-18T04:20:31.7462223Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12858
2022-05-18T04:20:31.7486521Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12859
2022-05-18T04:20:32.5626871Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:20:32.5728154Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:20:32.5728890Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:20:32.5729672Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:20:32.5730200Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:20:32.5730720Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:20:32.5738255Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:20:32.5740288Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:20:32.5740709Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:20:32.5741081Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:20:32.5845744Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-05-18T04:20:32.5846907Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:20:32.5847494Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:20:32.5944343Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:20:32.5947676Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:20:32.9540080Z ok (1.494s)
2022-05-18T04:20:32.9540290Z 
2022-05-18T04:20:32.9540849Z ----------------------------------------------------------------------
2022-05-18T04:20:32.9541159Z Ran 1 test in 1.494s
2022-05-18T04:20:32.9541274Z 
2022-05-18T04:20:32.9541336Z OK
2022-05-18T04:20:32.9541444Z 
2022-05-18T04:20:32.9541527Z Generating XML reports...
2022-05-18T04:20:32.9572211Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042031.xml
2022-05-18T04:20:33.8723107Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:20:33.8733542Z 
2022-05-18T04:20:33.8733630Z Running tests...
2022-05-18T04:20:33.8734391Z ----------------------------------------------------------------------
2022-05-18T04:20:34.1543160Z   test_all_reduce_coalesced_max (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12918
2022-05-18T04:20:34.1566333Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12919
2022-05-18T04:20:34.1589977Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12920
2022-05-18T04:20:34.9591787Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:20:34.9652067Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:20:34.9652617Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:20:34.9653250Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:20:34.9653770Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:20:34.9692954Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:20:34.9760631Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:20:34.9761434Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:20:35.0703856Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:20:35.3644580Z ok (1.491s)
2022-05-18T04:20:35.3644799Z 
2022-05-18T04:20:35.3645312Z ----------------------------------------------------------------------
2022-05-18T04:20:35.3645777Z Ran 1 test in 1.491s
2022-05-18T04:20:35.3645894Z 
2022-05-18T04:20:35.3645956Z OK
2022-05-18T04:20:35.3646046Z 
2022-05-18T04:20:35.3646125Z Generating XML reports...
2022-05-18T04:20:35.3675280Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042033.xml
2022-05-18T04:20:36.2868288Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:20:36.2877545Z 
2022-05-18T04:20:36.2877681Z Running tests...
2022-05-18T04:20:36.2878124Z ----------------------------------------------------------------------
2022-05-18T04:20:36.5707905Z   test_all_reduce_coalesced_max_complex_unsupported (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12974
2022-05-18T04:20:36.5729355Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12975
2022-05-18T04:20:36.5753345Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12976
2022-05-18T04:20:37.3790137Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:20:37.3890916Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:20:37.3891533Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:20:37.3892573Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:20:37.3893420Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:20:37.3894056Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:20:37.3902351Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:20:37.3902799Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:20:37.3904064Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:20:37.5803696Z ok (1.292s)
2022-05-18T04:20:37.5804161Z 
2022-05-18T04:20:37.5804677Z ----------------------------------------------------------------------
2022-05-18T04:20:37.5804950Z Ran 1 test in 1.293s
2022-05-18T04:20:37.5805077Z 
2022-05-18T04:20:37.5805138Z OK
2022-05-18T04:20:37.5805250Z 
2022-05-18T04:20:37.5805391Z Generating XML reports...
2022-05-18T04:20:37.5836878Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042036.xml
2022-05-18T04:20:38.4984976Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:20:38.4995697Z 
2022-05-18T04:20:38.4995987Z Running tests...
2022-05-18T04:20:38.4996907Z ----------------------------------------------------------------------
2022-05-18T04:20:38.7812253Z   test_all_reduce_coalesced_min (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13027
2022-05-18T04:20:38.7835009Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13028
2022-05-18T04:20:38.7858697Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13029
2022-05-18T04:20:39.5805807Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:20:39.5906336Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:20:39.5906928Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:20:39.5907882Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:20:39.5908818Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:20:39.5909729Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:20:39.6015591Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:20:39.6016028Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:20:39.6918393Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:20:39.9910696Z ok (1.491s)
2022-05-18T04:20:39.9910945Z 
2022-05-18T04:20:39.9911473Z ----------------------------------------------------------------------
2022-05-18T04:20:39.9911924Z Ran 1 test in 1.491s
2022-05-18T04:20:39.9912041Z 
2022-05-18T04:20:39.9912109Z OK
2022-05-18T04:20:39.9912204Z 
2022-05-18T04:20:39.9912301Z Generating XML reports...
2022-05-18T04:20:39.9944726Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042038.xml
2022-05-18T04:20:40.9182278Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:20:40.9192334Z 
2022-05-18T04:20:40.9192467Z Running tests...
2022-05-18T04:20:40.9193024Z ----------------------------------------------------------------------
2022-05-18T04:20:41.2095688Z   test_all_reduce_coalesced_product (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13083
2022-05-18T04:20:41.2118458Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13084
2022-05-18T04:20:41.2142221Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13085
2022-05-18T04:20:41.9982313Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:20:41.9983153Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:20:41.9983526Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:20:41.9984155Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:20:41.9984670Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:20:41.9985195Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:20:42.0088888Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:20:42.0995855Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:20:42.0996223Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:20:42.4196569Z ok (1.500s)
2022-05-18T04:20:42.4196863Z 
2022-05-18T04:20:42.4197357Z ----------------------------------------------------------------------
2022-05-18T04:20:42.4197736Z Ran 1 test in 1.500s
2022-05-18T04:20:42.4197889Z 
2022-05-18T04:20:42.4197958Z OK
2022-05-18T04:20:42.4198038Z 
2022-05-18T04:20:42.4198132Z Generating XML reports...
2022-05-18T04:20:42.4228312Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042040.xml
2022-05-18T04:20:43.3481767Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:20:43.3491162Z 
2022-05-18T04:20:43.3491298Z Running tests...
2022-05-18T04:20:43.3491872Z ----------------------------------------------------------------------
2022-05-18T04:20:43.6322012Z   test_all_reduce_coalesced_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13139
2022-05-18T04:20:43.6345430Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13140
2022-05-18T04:20:43.6369537Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13141
2022-05-18T04:20:44.4193689Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:20:44.4294537Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:20:44.4295250Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:20:44.4296316Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:20:44.4296975Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:20:44.4297522Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:20:44.4304421Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:20:44.4305550Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:20:44.4306369Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:20:44.7421855Z ok (1.393s)
2022-05-18T04:20:44.7422085Z 
2022-05-18T04:20:44.7422547Z ----------------------------------------------------------------------
2022-05-18T04:20:44.7423083Z Ran 1 test in 1.393s
2022-05-18T04:20:44.7423265Z 
2022-05-18T04:20:44.7423357Z OK
2022-05-18T04:20:44.7423499Z 
2022-05-18T04:20:44.7423646Z Generating XML reports...
2022-05-18T04:20:44.7455282Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042043.xml
2022-05-18T04:20:45.6674167Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:20:45.6684121Z 
2022-05-18T04:20:45.6684240Z Running tests...
2022-05-18T04:20:45.6684818Z ----------------------------------------------------------------------
2022-05-18T04:20:45.9539251Z   test_all_reduce_complex_unsupported_ops (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13195
2022-05-18T04:20:45.9561695Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13196
2022-05-18T04:20:45.9585264Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13197
2022-05-18T04:20:46.7504128Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:20:46.7529835Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:20:46.7530451Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:20:46.7531135Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:20:46.7531667Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:20:46.7605108Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:20:46.7638856Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:20:46.7639567Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:20:46.8616296Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:20:47.0638620Z ok (1.395s)
2022-05-18T04:20:47.0639049Z 
2022-05-18T04:20:47.0639557Z ----------------------------------------------------------------------
2022-05-18T04:20:47.0639902Z Ran 1 test in 1.395s
2022-05-18T04:20:47.0640019Z 
2022-05-18T04:20:47.0640077Z OK
2022-05-18T04:20:47.0640172Z 
2022-05-18T04:20:47.0640267Z Generating XML reports...
2022-05-18T04:20:47.0670389Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042045.xml
2022-05-18T04:20:47.9960617Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:20:47.9970577Z 
2022-05-18T04:20:47.9970658Z Running tests...
2022-05-18T04:20:47.9971166Z ----------------------------------------------------------------------
2022-05-18T04:20:48.2783999Z   test_all_reduce_full_group_max (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13248
2022-05-18T04:20:48.2806120Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13249
2022-05-18T04:20:48.2828615Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13250
2022-05-18T04:20:49.1068998Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:20:49.1069623Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:20:49.1070216Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:20:49.1070854Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:20:49.1071384Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:20:49.1071929Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:20:49.1079166Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:20:49.1080740Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:20:49.1081839Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:20:49.1187914Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:20:49.1290122Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-05-18T04:20:49.1290771Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:20:49.1291695Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:20:49.1292324Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:20:49.1293037Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:20:49.3879555Z ok (1.391s)
2022-05-18T04:20:49.3879790Z 
2022-05-18T04:20:49.3880643Z ----------------------------------------------------------------------
2022-05-18T04:20:49.3880941Z Ran 1 test in 1.391s
2022-05-18T04:20:49.3881059Z 
2022-05-18T04:20:49.3881121Z OK
2022-05-18T04:20:49.3881212Z 
2022-05-18T04:20:49.3881311Z Generating XML reports...
2022-05-18T04:20:49.3910965Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042047.xml
2022-05-18T04:20:50.2931084Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:20:50.2940834Z 
2022-05-18T04:20:50.2941328Z Running tests...
2022-05-18T04:20:50.2941733Z ----------------------------------------------------------------------
2022-05-18T04:20:50.5751388Z   test_all_reduce_full_group_min (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13313
2022-05-18T04:20:50.5773672Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13314
2022-05-18T04:20:50.5797609Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13315
2022-05-18T04:20:51.3834779Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:20:51.3835221Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:20:51.3835669Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:20:51.3836319Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:20:51.3836880Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:20:51.3837683Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:20:51.3845030Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:20:51.3845858Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:20:51.3846938Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:20:51.4051475Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:20:51.4152463Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:20:51.4153350Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-05-18T04:20:51.4154543Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:20:51.4155628Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:20:51.4156477Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:20:51.6848539Z ok (1.391s)
2022-05-18T04:20:51.6848844Z 
2022-05-18T04:20:51.6849269Z ----------------------------------------------------------------------
2022-05-18T04:20:51.6849650Z Ran 1 test in 1.391s
2022-05-18T04:20:51.6849863Z 
2022-05-18T04:20:51.6849983Z OK
2022-05-18T04:20:51.6850151Z 
2022-05-18T04:20:51.6850307Z Generating XML reports...
2022-05-18T04:20:51.6881150Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042050.xml
2022-05-18T04:20:52.6061053Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:20:52.6071870Z 
2022-05-18T04:20:52.6072116Z Running tests...
2022-05-18T04:20:52.6072761Z ----------------------------------------------------------------------
2022-05-18T04:20:52.8911577Z   test_all_reduce_full_group_product (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13378
2022-05-18T04:20:52.8933109Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13379
2022-05-18T04:20:52.8957605Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13380
2022-05-18T04:20:53.6977054Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:20:53.7077816Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:20:53.7078489Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:20:53.7079093Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:20:53.7079637Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:20:53.7080165Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:20:53.7187383Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:20:53.7187788Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:20:53.8089855Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:20:53.8300556Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:20:53.8402091Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:20:53.8402707Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-05-18T04:20:53.8403785Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:20:53.8404489Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:20:53.8405071Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:20:54.1009700Z ok (1.493s)
2022-05-18T04:20:54.1009971Z 
2022-05-18T04:20:54.1010437Z ----------------------------------------------------------------------
2022-05-18T04:20:54.1010704Z Ran 1 test in 1.494s
2022-05-18T04:20:54.1010820Z 
2022-05-18T04:20:54.1010887Z OK
2022-05-18T04:20:54.1010969Z 
2022-05-18T04:20:54.1011063Z Generating XML reports...
2022-05-18T04:20:54.1042390Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042052.xml
2022-05-18T04:20:55.0212048Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:20:55.0221671Z 
2022-05-18T04:20:55.0221774Z Running tests...
2022-05-18T04:20:55.0222733Z ----------------------------------------------------------------------
2022-05-18T04:20:55.3075742Z   test_all_reduce_full_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13443
2022-05-18T04:20:55.3097701Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13444
2022-05-18T04:20:55.3121529Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13445
2022-05-18T04:20:56.1058207Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:20:56.1159167Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:20:56.1160091Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:20:56.1161387Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:20:56.1161960Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:20:56.1162484Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:20:56.1170015Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:20:56.1170656Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:20:56.1171516Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:20:56.1376266Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:20:56.1376733Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:20:56.1377178Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-05-18T04:20:56.1377832Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:20:56.1378367Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:20:56.1379051Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:20:56.4173415Z ok (1.395s)
2022-05-18T04:20:56.4173638Z 
2022-05-18T04:20:56.4174053Z ----------------------------------------------------------------------
2022-05-18T04:20:56.4174309Z Ran 1 test in 1.395s
2022-05-18T04:20:56.4174470Z 
2022-05-18T04:20:56.4175290Z OK
2022-05-18T04:20:56.4175500Z 
2022-05-18T04:20:56.4175599Z Generating XML reports...
2022-05-18T04:20:56.4206740Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042055.xml
2022-05-18T04:20:57.3444185Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:20:57.3454268Z 
2022-05-18T04:20:57.3454371Z Running tests...
2022-05-18T04:20:57.3454947Z ----------------------------------------------------------------------
2022-05-18T04:20:57.6311766Z   test_all_reduce_group_max (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13508
2022-05-18T04:20:57.6334040Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13509
2022-05-18T04:20:57.6357129Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13510
2022-05-18T04:20:58.4521042Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:20:58.4623239Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:20:58.4624212Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:20:58.4624622Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:20:58.4625109Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:20:58.4625625Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:20:58.4634108Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:20:58.4634698Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:20:58.4636081Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:20:58.4636563Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:20:58.4842441Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-05-18T04:20:58.4842863Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:20:58.4843531Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:20:58.4844112Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:20:58.4940492Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:20:58.8411906Z ok (1.495s)
2022-05-18T04:20:58.8412117Z 
2022-05-18T04:20:58.8412669Z ----------------------------------------------------------------------
2022-05-18T04:20:58.8413037Z Ran 1 test in 1.496s
2022-05-18T04:20:58.8413151Z 
2022-05-18T04:20:58.8413199Z OK
2022-05-18T04:20:58.8413297Z 
2022-05-18T04:20:58.8413394Z Generating XML reports...
2022-05-18T04:20:58.8444251Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042057.xml
2022-05-18T04:20:59.7751449Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:20:59.7761442Z 
2022-05-18T04:20:59.7761562Z Running tests...
2022-05-18T04:20:59.7762059Z ----------------------------------------------------------------------
2022-05-18T04:21:00.0591067Z   test_all_reduce_group_min (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13569
2022-05-18T04:21:00.0613656Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13570
2022-05-18T04:21:00.0637724Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13571
2022-05-18T04:21:00.8744467Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:21:00.8845535Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:21:00.8846043Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:21:00.8846673Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:21:00.8847426Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:21:00.8848296Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:21:00.8952564Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:21:00.9858574Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:21:00.9885942Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:21:00.9889598Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:21:00.9963072Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-05-18T04:21:00.9965619Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:21:00.9966266Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:21:01.0064342Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:21:01.0065327Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:21:01.3695614Z ok (1.593s)
2022-05-18T04:21:01.3695855Z 
2022-05-18T04:21:01.3696612Z ----------------------------------------------------------------------
2022-05-18T04:21:01.3696869Z Ran 1 test in 1.593s
2022-05-18T04:21:01.3696985Z 
2022-05-18T04:21:01.3697047Z OK
2022-05-18T04:21:01.3697138Z 
2022-05-18T04:21:01.3697218Z Generating XML reports...
2022-05-18T04:21:01.3729464Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042059.xml
2022-05-18T04:21:02.3127478Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:21:02.3137843Z 
2022-05-18T04:21:02.3138066Z Running tests...
2022-05-18T04:21:02.3138475Z ----------------------------------------------------------------------
2022-05-18T04:21:02.5981891Z   test_all_reduce_group_product (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13630
2022-05-18T04:21:02.6004138Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13631
2022-05-18T04:21:02.6028495Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13632
2022-05-18T04:21:03.4076185Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:21:03.4177357Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:21:03.4177826Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:21:03.4178535Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:21:03.4179067Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:21:03.4179603Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:21:03.4286328Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:21:03.5192012Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:21:03.5192539Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:21:03.5193021Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:21:03.5397407Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-05-18T04:21:03.5398120Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:21:03.5398785Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:21:03.5399334Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:21:03.5497508Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:21:03.9084309Z ok (1.594s)
2022-05-18T04:21:03.9084558Z 
2022-05-18T04:21:03.9085063Z ----------------------------------------------------------------------
2022-05-18T04:21:03.9085498Z Ran 1 test in 1.595s
2022-05-18T04:21:03.9085722Z 
2022-05-18T04:21:03.9085831Z OK
2022-05-18T04:21:03.9085996Z 
2022-05-18T04:21:03.9086159Z Generating XML reports...
2022-05-18T04:21:03.9116131Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042102.xml
2022-05-18T04:21:04.8318576Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:21:04.8327674Z 
2022-05-18T04:21:04.8328122Z Running tests...
2022-05-18T04:21:04.8328503Z ----------------------------------------------------------------------
2022-05-18T04:21:05.1177398Z   test_all_reduce_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13691
2022-05-18T04:21:05.1200200Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13692
2022-05-18T04:21:05.1222189Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13693
2022-05-18T04:21:05.9636553Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:21:05.9690512Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:21:05.9690971Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:21:05.9691578Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:21:05.9692180Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:21:05.9737750Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:21:05.9799080Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:21:05.9801103Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:21:06.0007970Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-05-18T04:21:06.0008366Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:21:06.0749131Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:21:06.0751046Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:21:06.0752082Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:21:06.0816016Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:21:06.0816546Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:21:06.4276759Z ok (1.595s)
2022-05-18T04:21:06.4276987Z 
2022-05-18T04:21:06.4277509Z ----------------------------------------------------------------------
2022-05-18T04:21:06.4277884Z Ran 1 test in 1.595s
2022-05-18T04:21:06.4278007Z 
2022-05-18T04:21:06.4278072Z OK
2022-05-18T04:21:06.4278163Z 
2022-05-18T04:21:06.4278253Z Generating XML reports...
2022-05-18T04:21:06.4308514Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042104.xml
2022-05-18T04:21:07.3547763Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:21:07.3558029Z 
2022-05-18T04:21:07.3558126Z Running tests...
2022-05-18T04:21:07.3558529Z ----------------------------------------------------------------------
2022-05-18T04:21:07.6423271Z   test_all_reduce_max (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13752
2022-05-18T04:21:07.6445192Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13753
2022-05-18T04:21:07.6469546Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13754
2022-05-18T04:21:08.4477725Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:21:08.4578622Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:21:08.4579300Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:21:08.4580536Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:21:08.4581150Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:21:08.4581673Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:21:08.4590335Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:21:08.4590929Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:21:08.4591457Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:21:08.7520513Z ok (1.396s)
2022-05-18T04:21:08.7520730Z 
2022-05-18T04:21:08.7521262Z ----------------------------------------------------------------------
2022-05-18T04:21:08.7521704Z Ran 1 test in 1.396s
2022-05-18T04:21:08.7521861Z 
2022-05-18T04:21:08.7521930Z OK
2022-05-18T04:21:08.7522024Z 
2022-05-18T04:21:08.7522127Z Generating XML reports...
2022-05-18T04:21:08.7552773Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042107.xml
2022-05-18T04:21:09.6861176Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:21:09.6871930Z 
2022-05-18T04:21:09.6872003Z Running tests...
2022-05-18T04:21:09.6873043Z ----------------------------------------------------------------------
2022-05-18T04:21:09.9731054Z   test_all_reduce_min (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13808
2022-05-18T04:21:09.9753496Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13809
2022-05-18T04:21:09.9777375Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13810
2022-05-18T04:21:10.7512621Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:21:10.7579189Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:21:10.7579786Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:21:10.7580428Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:21:10.7580946Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:21:10.7612906Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:21:10.7688054Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:21:10.7688786Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:21:10.8624844Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:21:11.1830885Z ok (1.496s)
2022-05-18T04:21:11.1831136Z 
2022-05-18T04:21:11.1831610Z ----------------------------------------------------------------------
2022-05-18T04:21:11.1831990Z Ran 1 test in 1.496s
2022-05-18T04:21:11.1832173Z 
2022-05-18T04:21:11.1832278Z OK
2022-05-18T04:21:11.1832406Z 
2022-05-18T04:21:11.1832555Z Generating XML reports...
2022-05-18T04:21:11.1864176Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042109.xml
2022-05-18T04:21:12.1086831Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:21:12.1096985Z 
2022-05-18T04:21:12.1097084Z Running tests...
2022-05-18T04:21:12.1098038Z ----------------------------------------------------------------------
2022-05-18T04:21:12.3976783Z   test_all_reduce_multigpu (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13864
2022-05-18T04:21:12.3998239Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13865
2022-05-18T04:21:12.4020978Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13866
2022-05-18T04:21:13.2167325Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:21:13.2267441Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:21:13.2268042Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:21:13.2268938Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:21:13.2269726Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:21:13.2272029Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:21:13.2378472Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:21:13.2378973Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:21:13.3283482Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:21:13.5071234Z skip: CUDA is not available. (1.397s)
2022-05-18T04:21:13.5071579Z 
2022-05-18T04:21:13.5071978Z ----------------------------------------------------------------------
2022-05-18T04:21:13.5072256Z Ran 1 test in 1.397s
2022-05-18T04:21:13.5072358Z 
2022-05-18T04:21:13.5072432Z OK (skipped=1)
2022-05-18T04:21:13.5072559Z 
2022-05-18T04:21:13.5072690Z Generating XML reports...
2022-05-18T04:21:13.5103146Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042112.xml
2022-05-18T04:21:14.4296874Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:21:14.4306401Z 
2022-05-18T04:21:14.4306523Z Running tests...
2022-05-18T04:21:14.4307122Z ----------------------------------------------------------------------
2022-05-18T04:21:14.7127182Z   test_all_reduce_multigpu_complex (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13917
2022-05-18T04:21:14.7149985Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13918
2022-05-18T04:21:14.7174321Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13919
2022-05-18T04:21:15.5122906Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:21:15.5224829Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:21:15.5225588Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:21:15.5226607Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:21:15.5227139Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:21:15.5227663Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:21:15.5235063Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:21:15.5235634Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:21:15.5237514Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:21:15.7224642Z skip: CUDA is not available. (1.291s)
2022-05-18T04:21:15.7224937Z 
2022-05-18T04:21:15.7225429Z ----------------------------------------------------------------------
2022-05-18T04:21:15.7225856Z Ran 1 test in 1.292s
2022-05-18T04:21:15.7225976Z 
2022-05-18T04:21:15.7226049Z OK (skipped=1)
2022-05-18T04:21:15.7226158Z 
2022-05-18T04:21:15.7226243Z Generating XML reports...
2022-05-18T04:21:15.7255924Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042114.xml
2022-05-18T04:21:16.6485568Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:21:16.6495401Z 
2022-05-18T04:21:16.6495847Z Running tests...
2022-05-18T04:21:16.6496303Z ----------------------------------------------------------------------
2022-05-18T04:21:16.9343278Z   test_all_reduce_product (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13970
2022-05-18T04:21:16.9365732Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13971
2022-05-18T04:21:16.9388797Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13972
2022-05-18T04:21:17.7489476Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:21:17.7537133Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:21:17.7537581Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:21:17.7538272Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:21:17.7538853Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:21:17.7591373Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:21:17.7646955Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:21:17.7647880Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:21:17.8602639Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:21:18.1442487Z ok (1.494s)
2022-05-18T04:21:18.1442739Z 
2022-05-18T04:21:18.1443275Z ----------------------------------------------------------------------
2022-05-18T04:21:18.1443569Z Ran 1 test in 1.495s
2022-05-18T04:21:18.1443682Z 
2022-05-18T04:21:18.1443743Z OK
2022-05-18T04:21:18.1443833Z 
2022-05-18T04:21:18.1443926Z Generating XML reports...
2022-05-18T04:21:18.1474502Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042116.xml
2022-05-18T04:21:19.0649761Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:21:19.0659256Z 
2022-05-18T04:21:19.0659388Z Running tests...
2022-05-18T04:21:19.0659826Z ----------------------------------------------------------------------
2022-05-18T04:21:19.3489988Z   test_all_reduce_result_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14026
2022-05-18T04:21:19.3513198Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14027
2022-05-18T04:21:19.3537297Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14028
2022-05-18T04:21:20.1530982Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:21:20.1531708Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:21:20.1532389Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:21:20.1533288Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:21:20.1534095Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:21:20.1534626Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:21:20.1541187Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:21:20.1542015Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:21:20.1542677Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:21:20.3589733Z skip: CUDA is not available. (1.293s)
2022-05-18T04:21:20.3589930Z 
2022-05-18T04:21:20.3590237Z ----------------------------------------------------------------------
2022-05-18T04:21:20.3590483Z Ran 1 test in 1.293s
2022-05-18T04:21:20.3590619Z 
2022-05-18T04:21:20.3590690Z OK (skipped=1)
2022-05-18T04:21:20.3590798Z 
2022-05-18T04:21:20.3590871Z Generating XML reports...
2022-05-18T04:21:20.3620407Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042119.xml
2022-05-18T04:21:21.2834194Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:21:21.2844509Z 
2022-05-18T04:21:21.2844647Z Running tests...
2022-05-18T04:21:21.2845079Z ----------------------------------------------------------------------
2022-05-18T04:21:21.5664592Z   test_all_reduce_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14079
2022-05-18T04:21:21.5688701Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14080
2022-05-18T04:21:21.5712050Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14081
2022-05-18T04:21:22.4103673Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:21:22.4104410Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:21:22.4104918Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:21:22.4105861Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:21:22.4106740Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:21:22.4107425Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:21:22.4113691Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:21:22.4114809Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:21:22.4116094Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:21:22.6764803Z ok (1.392s)
2022-05-18T04:21:22.6765028Z 
2022-05-18T04:21:22.6765529Z ----------------------------------------------------------------------
2022-05-18T04:21:22.6765949Z Ran 1 test in 1.392s
2022-05-18T04:21:22.6766065Z 
2022-05-18T04:21:22.6766133Z OK
2022-05-18T04:21:22.6766224Z 
2022-05-18T04:21:22.6766300Z Generating XML reports...
2022-05-18T04:21:22.6796738Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042121.xml
2022-05-18T04:21:23.5967052Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:21:23.5977495Z 
2022-05-18T04:21:23.5977600Z Running tests...
2022-05-18T04:21:23.5978039Z ----------------------------------------------------------------------
2022-05-18T04:21:23.8839661Z   test_all_reduce_sum_async (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14135
2022-05-18T04:21:23.8862124Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14136
2022-05-18T04:21:23.8886401Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14137
2022-05-18T04:21:24.7027352Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:21:24.7128281Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:21:24.7128867Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:21:24.7129670Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:21:24.7130200Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:21:24.7130745Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:21:24.7236367Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:21:24.8142751Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:21:24.8143670Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:21:25.0939229Z ok (1.496s)
2022-05-18T04:21:25.0939488Z 
2022-05-18T04:21:25.0939987Z ----------------------------------------------------------------------
2022-05-18T04:21:25.0940444Z Ran 1 test in 1.496s
2022-05-18T04:21:25.0940652Z 
2022-05-18T04:21:25.0940762Z OK
2022-05-18T04:21:25.0940926Z 
2022-05-18T04:21:25.0941093Z Generating XML reports...
2022-05-18T04:21:25.0971494Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042123.xml
2022-05-18T04:21:26.0319392Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:21:26.0329151Z 
2022-05-18T04:21:26.0329414Z Running tests...
2022-05-18T04:21:26.0329772Z ----------------------------------------------------------------------
2022-05-18T04:21:26.3192207Z   test_all_reduce_sum_complex (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14191
2022-05-18T04:21:26.3214936Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14192
2022-05-18T04:21:26.3238612Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14193
2022-05-18T04:21:27.1552546Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:21:27.1553063Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:21:27.1553677Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:21:27.1554116Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:21:27.1554601Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:21:27.1555113Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:21:27.1660608Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:21:27.1661482Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:21:27.2565745Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:21:27.5293694Z ok (1.496s)
2022-05-18T04:21:27.5293941Z 
2022-05-18T04:21:27.5294388Z ----------------------------------------------------------------------
2022-05-18T04:21:27.5295028Z Ran 1 test in 1.496s
2022-05-18T04:21:27.5295208Z 
2022-05-18T04:21:27.5295305Z OK
2022-05-18T04:21:27.5295464Z 
2022-05-18T04:21:27.5295700Z Generating XML reports...
2022-05-18T04:21:27.5327560Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042126.xml
2022-05-18T04:21:28.4695904Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:21:28.4705772Z 
2022-05-18T04:21:28.4705908Z Running tests...
2022-05-18T04:21:28.4706327Z ----------------------------------------------------------------------
2022-05-18T04:21:28.7555225Z   test_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14247
2022-05-18T04:21:28.7577781Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14248
2022-05-18T04:21:28.7602304Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14249
2022-05-18T04:21:29.5741253Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:21:29.5842557Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:21:29.5843711Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:21:29.5844442Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:21:29.5844932Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:21:29.5845452Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:21:29.5852739Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:21:29.5853295Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:21:29.5854517Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:21:29.7653063Z skip: CUDA is not available. (1.294s)
2022-05-18T04:21:29.7653272Z 
2022-05-18T04:21:29.7653722Z ----------------------------------------------------------------------
2022-05-18T04:21:29.7653976Z Ran 1 test in 1.295s
2022-05-18T04:21:29.7654089Z 
2022-05-18T04:21:29.7654162Z OK (skipped=1)
2022-05-18T04:21:29.7654258Z 
2022-05-18T04:21:29.7654344Z Generating XML reports...
2022-05-18T04:21:29.7686991Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042128.xml
2022-05-18T04:21:30.6908934Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:21:30.6919279Z 
2022-05-18T04:21:30.6919415Z Running tests...
2022-05-18T04:21:30.6919880Z ----------------------------------------------------------------------
2022-05-18T04:21:30.9757184Z   test_all_reduce_sum_cuda_async (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14300
2022-05-18T04:21:30.9778527Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14301
2022-05-18T04:21:30.9803877Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14302
2022-05-18T04:21:31.7610208Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:21:31.7610717Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:21:31.7611085Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:21:31.7611745Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:21:31.7612513Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:21:31.7613168Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:21:31.7620964Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:21:31.7621522Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:21:31.7623684Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:21:31.9853038Z skip: CUDA is not available. (1.293s)
2022-05-18T04:21:31.9853228Z 
2022-05-18T04:21:31.9853539Z ----------------------------------------------------------------------
2022-05-18T04:21:31.9853788Z Ran 1 test in 1.293s
2022-05-18T04:21:31.9853888Z 
2022-05-18T04:21:31.9853962Z OK (skipped=1)
2022-05-18T04:21:31.9854067Z 
2022-05-18T04:21:31.9854171Z Generating XML reports...
2022-05-18T04:21:31.9885476Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042130.xml
2022-05-18T04:21:32.9063681Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:21:32.9073392Z 
2022-05-18T04:21:32.9073938Z Running tests...
2022-05-18T04:21:32.9074341Z ----------------------------------------------------------------------
2022-05-18T04:21:33.1894710Z   test_all_reduce_sum_cuda_complex (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14353
2022-05-18T04:21:33.1917900Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14354
2022-05-18T04:21:33.1941748Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14355
2022-05-18T04:21:34.0258247Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:21:34.0346677Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:21:34.0347180Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:21:34.0347806Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:21:34.0348439Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:21:34.0359258Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:21:34.0456327Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:21:34.0456717Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:21:34.1372388Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:21:34.2992358Z skip: CUDA is not available. (1.392s)
2022-05-18T04:21:34.2992612Z 
2022-05-18T04:21:34.2992922Z ----------------------------------------------------------------------
2022-05-18T04:21:34.2993179Z Ran 1 test in 1.392s
2022-05-18T04:21:34.2993296Z 
2022-05-18T04:21:34.2993372Z OK (skipped=1)
2022-05-18T04:21:34.2993480Z 
2022-05-18T04:21:34.2993567Z Generating XML reports...
2022-05-18T04:21:34.3024463Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042132.xml
2022-05-18T04:21:35.2220947Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:21:35.2230998Z 
2022-05-18T04:21:35.2231135Z Running tests...
2022-05-18T04:21:35.2231725Z ----------------------------------------------------------------------
2022-05-18T04:21:35.2246584Z   test_all_to_all (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.001s)
2022-05-18T04:21:35.2247264Z 
2022-05-18T04:21:35.2247503Z ----------------------------------------------------------------------
2022-05-18T04:21:35.2247747Z Ran 1 test in 0.002s
2022-05-18T04:21:35.2247933Z 
2022-05-18T04:21:35.2248011Z OK (skipped=1)
2022-05-18T04:21:35.2248105Z 
2022-05-18T04:21:35.2248191Z Generating XML reports...
2022-05-18T04:21:35.2278863Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042135.xml
2022-05-18T04:21:36.0589791Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:21:36.0600030Z 
2022-05-18T04:21:36.0600510Z Running tests...
2022-05-18T04:21:36.0601106Z ----------------------------------------------------------------------
2022-05-18T04:21:36.0615454Z   test_all_to_all_complex (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.001s)
2022-05-18T04:21:36.0615800Z 
2022-05-18T04:21:36.0616176Z ----------------------------------------------------------------------
2022-05-18T04:21:36.0616501Z Ran 1 test in 0.002s
2022-05-18T04:21:36.0616604Z 
2022-05-18T04:21:36.0616678Z OK (skipped=1)
2022-05-18T04:21:36.0616787Z 
2022-05-18T04:21:36.0616882Z Generating XML reports...
2022-05-18T04:21:36.0647634Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042136.xml
2022-05-18T04:21:36.8918566Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:21:36.8928738Z 
2022-05-18T04:21:36.8928978Z Running tests...
2022-05-18T04:21:36.8929353Z ----------------------------------------------------------------------
2022-05-18T04:21:36.8945668Z   test_all_to_all_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only NCCL supports CUDA all_to_all (0.001s)
2022-05-18T04:21:36.8945919Z 
2022-05-18T04:21:36.8946226Z ----------------------------------------------------------------------
2022-05-18T04:21:36.8946502Z Ran 1 test in 0.002s
2022-05-18T04:21:36.8946616Z 
2022-05-18T04:21:36.8946691Z OK (skipped=1)
2022-05-18T04:21:36.8946799Z 
2022-05-18T04:21:36.8946888Z Generating XML reports...
2022-05-18T04:21:36.8985003Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042136.xml
2022-05-18T04:21:37.7304466Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:21:37.7314682Z 
2022-05-18T04:21:37.7314839Z Running tests...
2022-05-18T04:21:37.7315297Z ----------------------------------------------------------------------
2022-05-18T04:21:37.7331773Z   test_all_to_all_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only NCCL supports CUDA all_to_all (0.001s)
2022-05-18T04:21:37.7332027Z 
2022-05-18T04:21:37.7332270Z ----------------------------------------------------------------------
2022-05-18T04:21:37.7332504Z Ran 1 test in 0.002s
2022-05-18T04:21:37.7332620Z 
2022-05-18T04:21:37.7332709Z OK (skipped=1)
2022-05-18T04:21:37.7332820Z 
2022-05-18T04:21:37.7332907Z Generating XML reports...
2022-05-18T04:21:37.7364110Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042137.xml
2022-05-18T04:21:38.5641010Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:21:38.5651273Z 
2022-05-18T04:21:38.5651378Z Running tests...
2022-05-18T04:21:38.5651947Z ----------------------------------------------------------------------
2022-05-18T04:21:38.5666892Z   test_all_to_all_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.001s)
2022-05-18T04:21:38.5667241Z 
2022-05-18T04:21:38.5667572Z ----------------------------------------------------------------------
2022-05-18T04:21:38.5667825Z Ran 1 test in 0.002s
2022-05-18T04:21:38.5667943Z 
2022-05-18T04:21:38.5668019Z OK (skipped=1)
2022-05-18T04:21:38.5668133Z 
2022-05-18T04:21:38.5668255Z Generating XML reports...
2022-05-18T04:21:38.5698761Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042138.xml
2022-05-18T04:21:39.3975656Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:21:39.3985703Z 
2022-05-18T04:21:39.3985968Z Running tests...
2022-05-18T04:21:39.3986603Z ----------------------------------------------------------------------
2022-05-18T04:21:39.4002531Z   test_all_to_all_full_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only NCCL supports CUDA all_to_all (0.002s)
2022-05-18T04:21:39.4003054Z 
2022-05-18T04:21:39.4003573Z ----------------------------------------------------------------------
2022-05-18T04:21:39.4004003Z Ran 1 test in 0.002s
2022-05-18T04:21:39.4004200Z 
2022-05-18T04:21:39.4004325Z OK (skipped=1)
2022-05-18T04:21:39.4004486Z 
2022-05-18T04:21:39.4004630Z Generating XML reports...
2022-05-18T04:21:39.4035676Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042139.xml
2022-05-18T04:21:40.2339196Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:21:40.2350070Z 
2022-05-18T04:21:40.2350522Z Running tests...
2022-05-18T04:21:40.2350946Z ----------------------------------------------------------------------
2022-05-18T04:21:40.2365354Z   test_all_to_all_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.001s)
2022-05-18T04:21:40.2365788Z 
2022-05-18T04:21:40.2366268Z ----------------------------------------------------------------------
2022-05-18T04:21:40.2366720Z Ran 1 test in 0.002s
2022-05-18T04:21:40.2366946Z 
2022-05-18T04:21:40.2367074Z OK (skipped=1)
2022-05-18T04:21:40.2367201Z 
2022-05-18T04:21:40.2367287Z Generating XML reports...
2022-05-18T04:21:40.2398268Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042140.xml
2022-05-18T04:21:41.0708989Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:21:41.0719386Z 
2022-05-18T04:21:41.0719464Z Running tests...
2022-05-18T04:21:41.0720241Z ----------------------------------------------------------------------
2022-05-18T04:21:41.0736439Z   test_all_to_all_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-05-18T04:21:41.0736856Z 
2022-05-18T04:21:41.0737270Z ----------------------------------------------------------------------
2022-05-18T04:21:41.0737697Z Ran 1 test in 0.002s
2022-05-18T04:21:41.0737908Z 
2022-05-18T04:21:41.0738053Z OK (skipped=1)
2022-05-18T04:21:41.0738257Z 
2022-05-18T04:21:41.0738385Z Generating XML reports...
2022-05-18T04:21:41.0769372Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042141.xml
2022-05-18T04:21:41.9039446Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:21:41.9049572Z 
2022-05-18T04:21:41.9049669Z Running tests...
2022-05-18T04:21:41.9050129Z ----------------------------------------------------------------------
2022-05-18T04:21:41.9064999Z   test_all_to_all_single_equal_split (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.001s)
2022-05-18T04:21:41.9065360Z 
2022-05-18T04:21:41.9065666Z ----------------------------------------------------------------------
2022-05-18T04:21:41.9066005Z Ran 1 test in 0.002s
2022-05-18T04:21:41.9066159Z 
2022-05-18T04:21:41.9066233Z OK (skipped=1)
2022-05-18T04:21:41.9066344Z 
2022-05-18T04:21:41.9066429Z Generating XML reports...
2022-05-18T04:21:41.9106198Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042141.xml
2022-05-18T04:21:42.7467410Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:21:42.7477356Z 
2022-05-18T04:21:42.7477705Z Running tests...
2022-05-18T04:21:42.7478088Z ----------------------------------------------------------------------
2022-05-18T04:21:42.7494277Z   test_all_to_all_single_equal_split_complex (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-05-18T04:21:42.7494727Z 
2022-05-18T04:21:42.7495000Z ----------------------------------------------------------------------
2022-05-18T04:21:42.7495257Z Ran 1 test in 0.002s
2022-05-18T04:21:42.7495374Z 
2022-05-18T04:21:42.7495446Z OK (skipped=1)
2022-05-18T04:21:42.7495539Z 
2022-05-18T04:21:42.7495629Z Generating XML reports...
2022-05-18T04:21:42.7525990Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042142.xml
2022-05-18T04:21:43.5828019Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:21:43.5837833Z 
2022-05-18T04:21:43.5837938Z Running tests...
2022-05-18T04:21:43.5838517Z ----------------------------------------------------------------------
2022-05-18T04:21:43.5854196Z   test_all_to_all_single_equal_split_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-05-18T04:21:43.5854627Z 
2022-05-18T04:21:43.5854942Z ----------------------------------------------------------------------
2022-05-18T04:21:43.5855354Z Ran 1 test in 0.002s
2022-05-18T04:21:43.5855543Z 
2022-05-18T04:21:43.5855657Z OK (skipped=1)
2022-05-18T04:21:43.5855975Z 
2022-05-18T04:21:43.5856096Z Generating XML reports...
2022-05-18T04:21:43.5886939Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042143.xml
2022-05-18T04:21:44.4155129Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:21:44.4165231Z 
2022-05-18T04:21:44.4165367Z Running tests...
2022-05-18T04:21:44.4166522Z ----------------------------------------------------------------------
2022-05-18T04:21:44.4181730Z   test_all_to_all_single_equal_split_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.001s)
2022-05-18T04:21:44.4182153Z 
2022-05-18T04:21:44.4182550Z ----------------------------------------------------------------------
2022-05-18T04:21:44.4183181Z Ran 1 test in 0.002s
2022-05-18T04:21:44.4183382Z 
2022-05-18T04:21:44.4183474Z OK (skipped=1)
2022-05-18T04:21:44.4183589Z 
2022-05-18T04:21:44.4183675Z Generating XML reports...
2022-05-18T04:21:44.4214233Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042144.xml
2022-05-18T04:21:45.2476531Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:21:45.2486752Z 
2022-05-18T04:21:45.2486850Z Running tests...
2022-05-18T04:21:45.2487395Z ----------------------------------------------------------------------
2022-05-18T04:21:45.2502062Z   test_all_to_all_single_equal_split_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.001s)
2022-05-18T04:21:45.2502461Z 
2022-05-18T04:21:45.2502779Z ----------------------------------------------------------------------
2022-05-18T04:21:45.2503221Z Ran 1 test in 0.002s
2022-05-18T04:21:45.2503340Z 
2022-05-18T04:21:45.2503416Z OK (skipped=1)
2022-05-18T04:21:45.2503523Z 
2022-05-18T04:21:45.2503608Z Generating XML reports...
2022-05-18T04:21:45.2534488Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042145.xml
2022-05-18T04:21:46.0822151Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:21:46.0832426Z 
2022-05-18T04:21:46.0832530Z Running tests...
2022-05-18T04:21:46.0832949Z ----------------------------------------------------------------------
2022-05-18T04:21:46.0849454Z   test_all_to_all_single_equal_split_full_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-05-18T04:21:46.0850255Z 
2022-05-18T04:21:46.0850540Z ----------------------------------------------------------------------
2022-05-18T04:21:46.0850859Z Ran 1 test in 0.002s
2022-05-18T04:21:46.0850962Z 
2022-05-18T04:21:46.0851035Z OK (skipped=1)
2022-05-18T04:21:46.0851143Z 
2022-05-18T04:21:46.0851228Z Generating XML reports...
2022-05-18T04:21:46.0882120Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042146.xml
2022-05-18T04:21:46.9224363Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:21:46.9234419Z 
2022-05-18T04:21:46.9234550Z Running tests...
2022-05-18T04:21:46.9235275Z ----------------------------------------------------------------------
2022-05-18T04:21:46.9250215Z   test_all_to_all_single_equal_split_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.001s)
2022-05-18T04:21:46.9250532Z 
2022-05-18T04:21:46.9250910Z ----------------------------------------------------------------------
2022-05-18T04:21:46.9251347Z Ran 1 test in 0.002s
2022-05-18T04:21:46.9251513Z 
2022-05-18T04:21:46.9251595Z OK (skipped=1)
2022-05-18T04:21:46.9251711Z 
2022-05-18T04:21:46.9251794Z Generating XML reports...
2022-05-18T04:21:46.9289081Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042146.xml
2022-05-18T04:21:47.7594357Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:21:47.7604318Z 
2022-05-18T04:21:47.7604478Z Running tests...
2022-05-18T04:21:47.7604910Z ----------------------------------------------------------------------
2022-05-18T04:21:47.7622274Z   test_all_to_all_single_equal_split_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-05-18T04:21:47.7622734Z 
2022-05-18T04:21:47.7623348Z ----------------------------------------------------------------------
2022-05-18T04:21:47.7623825Z Ran 1 test in 0.002s
2022-05-18T04:21:47.7624016Z 
2022-05-18T04:21:47.7624143Z OK (skipped=1)
2022-05-18T04:21:47.7624323Z 
2022-05-18T04:21:47.7624473Z Generating XML reports...
2022-05-18T04:21:47.7655697Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042147.xml
2022-05-18T04:21:48.5989838Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:21:48.6000410Z 
2022-05-18T04:21:48.6000678Z Running tests...
2022-05-18T04:21:48.6001139Z ----------------------------------------------------------------------
2022-05-18T04:21:48.6016485Z   test_all_to_all_single_unequal_split (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.001s)
2022-05-18T04:21:48.6017213Z 
2022-05-18T04:21:48.6017529Z ----------------------------------------------------------------------
2022-05-18T04:21:48.6017803Z Ran 1 test in 0.002s
2022-05-18T04:21:48.6017916Z 
2022-05-18T04:21:48.6017989Z OK (skipped=1)
2022-05-18T04:21:48.6018095Z 
2022-05-18T04:21:48.6018179Z Generating XML reports...
2022-05-18T04:21:48.6048678Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042148.xml
2022-05-18T04:21:49.4329626Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:21:49.4340145Z 
2022-05-18T04:21:49.4340617Z Running tests...
2022-05-18T04:21:49.4341015Z ----------------------------------------------------------------------
2022-05-18T04:21:49.4356478Z   test_all_to_all_single_unequal_split_complex (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.001s)
2022-05-18T04:21:49.4356992Z 
2022-05-18T04:21:49.4357527Z ----------------------------------------------------------------------
2022-05-18T04:21:49.4357817Z Ran 1 test in 0.002s
2022-05-18T04:21:49.4358175Z 
2022-05-18T04:21:49.4358255Z OK (skipped=1)
2022-05-18T04:21:49.4358366Z 
2022-05-18T04:21:49.4358454Z Generating XML reports...
2022-05-18T04:21:49.4388772Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042149.xml
2022-05-18T04:21:50.2694269Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:21:50.2704300Z 
2022-05-18T04:21:50.2704433Z Running tests...
2022-05-18T04:21:50.2705020Z ----------------------------------------------------------------------
2022-05-18T04:21:50.2720899Z   test_all_to_all_single_unequal_split_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.001s)
2022-05-18T04:21:50.2721232Z 
2022-05-18T04:21:50.2721527Z ----------------------------------------------------------------------
2022-05-18T04:21:50.2721821Z Ran 1 test in 0.002s
2022-05-18T04:21:50.2721935Z 
2022-05-18T04:21:50.2722011Z OK (skipped=1)
2022-05-18T04:21:50.2722140Z 
2022-05-18T04:21:50.2722219Z Generating XML reports...
2022-05-18T04:21:50.2753534Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042150.xml
2022-05-18T04:21:51.0966099Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:21:51.0975935Z 
2022-05-18T04:21:51.0976032Z Running tests...
2022-05-18T04:21:51.0976589Z ----------------------------------------------------------------------
2022-05-18T04:21:51.0993909Z   test_all_to_all_single_unequal_split_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-05-18T04:21:51.0994291Z 
2022-05-18T04:21:51.0994618Z ----------------------------------------------------------------------
2022-05-18T04:21:51.0994909Z Ran 1 test in 0.002s
2022-05-18T04:21:51.0995024Z 
2022-05-18T04:21:51.0995103Z OK (skipped=1)
2022-05-18T04:21:51.0995216Z 
2022-05-18T04:21:51.0995347Z Generating XML reports...
2022-05-18T04:21:51.1025989Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042151.xml
2022-05-18T04:21:51.9256377Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:21:51.9266562Z 
2022-05-18T04:21:51.9266731Z Running tests...
2022-05-18T04:21:51.9267161Z ----------------------------------------------------------------------
2022-05-18T04:21:51.9281556Z   test_all_to_all_single_unequal_split_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.001s)
2022-05-18T04:21:51.9282006Z 
2022-05-18T04:21:51.9282412Z ----------------------------------------------------------------------
2022-05-18T04:21:51.9282724Z Ran 1 test in 0.001s
2022-05-18T04:21:51.9282841Z 
2022-05-18T04:21:51.9282915Z OK (skipped=1)
2022-05-18T04:21:51.9283023Z 
2022-05-18T04:21:51.9283095Z Generating XML reports...
2022-05-18T04:21:51.9319682Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042151.xml
2022-05-18T04:21:52.7668874Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:21:52.7679710Z 
2022-05-18T04:21:52.7679984Z Running tests...
2022-05-18T04:21:52.7680458Z ----------------------------------------------------------------------
2022-05-18T04:21:52.7696297Z   test_all_to_all_single_unequal_split_full_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-05-18T04:21:52.7696644Z 
2022-05-18T04:21:52.7697027Z ----------------------------------------------------------------------
2022-05-18T04:21:52.7697451Z Ran 1 test in 0.002s
2022-05-18T04:21:52.7697645Z 
2022-05-18T04:21:52.7697766Z OK (skipped=1)
2022-05-18T04:21:52.7697901Z 
2022-05-18T04:21:52.7697987Z Generating XML reports...
2022-05-18T04:21:52.7729348Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042152.xml
2022-05-18T04:21:53.6007082Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:21:53.6018128Z 
2022-05-18T04:21:53.6018670Z Running tests...
2022-05-18T04:21:53.6019042Z ----------------------------------------------------------------------
2022-05-18T04:21:53.6033629Z   test_all_to_all_single_unequal_split_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.001s)
2022-05-18T04:21:53.6034006Z 
2022-05-18T04:21:53.6034350Z ----------------------------------------------------------------------
2022-05-18T04:21:53.6034761Z Ran 1 test in 0.002s
2022-05-18T04:21:53.6034976Z 
2022-05-18T04:21:53.6035109Z OK (skipped=1)
2022-05-18T04:21:53.6035291Z 
2022-05-18T04:21:53.6035379Z Generating XML reports...
2022-05-18T04:21:53.6065436Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042153.xml
2022-05-18T04:21:54.4328465Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:21:54.4338540Z 
2022-05-18T04:21:54.4338688Z Running tests...
2022-05-18T04:21:54.4339864Z ----------------------------------------------------------------------
2022-05-18T04:21:54.4355214Z   test_all_to_all_single_unequal_split_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-05-18T04:21:54.4355617Z 
2022-05-18T04:21:54.4356030Z ----------------------------------------------------------------------
2022-05-18T04:21:54.4356413Z Ran 1 test in 0.002s
2022-05-18T04:21:54.4356528Z 
2022-05-18T04:21:54.4356588Z OK (skipped=1)
2022-05-18T04:21:54.4356696Z 
2022-05-18T04:21:54.4356780Z Generating XML reports...
2022-05-18T04:21:54.4387831Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042154.xml
2022-05-18T04:21:55.2689922Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:21:55.2699817Z 
2022-05-18T04:21:55.2699965Z Running tests...
2022-05-18T04:21:55.2700548Z ----------------------------------------------------------------------
2022-05-18T04:21:55.5562919Z   test_average_parameters (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14646
2022-05-18T04:21:55.5584602Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14647
2022-05-18T04:21:55.5609413Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14648
2022-05-18T04:21:56.3775351Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:21:56.3876527Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:21:56.3877506Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:21:56.3877934Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:21:56.3878432Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:21:56.3878944Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:21:56.3885961Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:21:56.3886506Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:21:56.3887306Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:21:56.5660633Z skip: Need at least 2 CUDA devices (1.296s)
2022-05-18T04:21:56.5660941Z 
2022-05-18T04:21:56.5661414Z ----------------------------------------------------------------------
2022-05-18T04:21:56.5661875Z Ran 1 test in 1.296s
2022-05-18T04:21:56.5661989Z 
2022-05-18T04:21:56.5662051Z OK (skipped=1)
2022-05-18T04:21:56.5662156Z 
2022-05-18T04:21:56.5662241Z Generating XML reports...
2022-05-18T04:21:56.5692593Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042155.xml
2022-05-18T04:21:57.4892087Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:21:57.4903371Z 
2022-05-18T04:21:57.4903659Z Running tests...
2022-05-18T04:21:57.4919275Z ----------------------------------------------------------------------
2022-05-18T04:21:57.4920222Z   test_backend_full_group (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'nccl'} (0.001s)
2022-05-18T04:21:57.4920633Z 
2022-05-18T04:21:57.4920980Z ----------------------------------------------------------------------
2022-05-18T04:21:57.4921371Z Ran 1 test in 0.002s
2022-05-18T04:21:57.4921582Z 
2022-05-18T04:21:57.4921704Z OK (skipped=1)
2022-05-18T04:21:57.4921884Z 
2022-05-18T04:21:57.4922029Z Generating XML reports...
2022-05-18T04:21:57.4953116Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042157.xml
2022-05-18T04:21:58.3236674Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:21:58.3246966Z 
2022-05-18T04:21:58.3247436Z Running tests...
2022-05-18T04:21:58.3247851Z ----------------------------------------------------------------------
2022-05-18T04:21:58.3263453Z   test_backend_group (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'nccl', 'gloo'} (0.001s)
2022-05-18T04:21:58.3263908Z 
2022-05-18T04:21:58.3264168Z ----------------------------------------------------------------------
2022-05-18T04:21:58.3264420Z Ran 1 test in 0.002s
2022-05-18T04:21:58.3264534Z 
2022-05-18T04:21:58.3264609Z OK (skipped=1)
2022-05-18T04:21:58.3264733Z 
2022-05-18T04:21:58.3264821Z Generating XML reports...
2022-05-18T04:21:58.3295766Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042158.xml
2022-05-18T04:21:59.1559195Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:21:59.1569633Z 
2022-05-18T04:21:59.1569912Z Running tests...
2022-05-18T04:21:59.1570314Z ----------------------------------------------------------------------
2022-05-18T04:21:59.4397647Z   test_barrier (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14719
2022-05-18T04:21:59.4419294Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14720
2022-05-18T04:21:59.4443960Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14721
2022-05-18T04:22:00.2249073Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:22:00.2350555Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:22:00.2351024Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:22:00.2351735Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:22:00.2352621Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:22:00.2353147Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:22:00.2361042Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:22:00.2361601Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:22:00.2362660Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:22:01.7512333Z ok (2.594s)
2022-05-18T04:22:01.7512644Z 
2022-05-18T04:22:01.7513225Z ----------------------------------------------------------------------
2022-05-18T04:22:01.7513516Z Ran 1 test in 2.594s
2022-05-18T04:22:01.7513666Z 
2022-05-18T04:22:01.7513757Z OK
2022-05-18T04:22:01.7513892Z 
2022-05-18T04:22:01.7514109Z Generating XML reports...
2022-05-18T04:22:01.7545410Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042159.xml
2022-05-18T04:22:02.6886866Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:22:02.6896975Z 
2022-05-18T04:22:02.6897080Z Running tests...
2022-05-18T04:22:02.6897649Z ----------------------------------------------------------------------
2022-05-18T04:22:02.9757685Z   test_barrier_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14772
2022-05-18T04:22:02.9779542Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14773
2022-05-18T04:22:02.9802977Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14774
2022-05-18T04:22:03.7695018Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:22:03.7695438Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:22:03.7695842Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:22:03.7696519Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:22:03.7697100Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:22:03.7697625Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:22:03.7801849Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:22:03.8710799Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:22:03.8711518Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:22:04.0852737Z skip: CUDA is not available. (1.395s)
2022-05-18T04:22:04.0852975Z 
2022-05-18T04:22:04.0853471Z ----------------------------------------------------------------------
2022-05-18T04:22:04.0853844Z Ran 1 test in 1.395s
2022-05-18T04:22:04.0853960Z 
2022-05-18T04:22:04.0854033Z OK (skipped=1)
2022-05-18T04:22:04.0854139Z 
2022-05-18T04:22:04.0854224Z Generating XML reports...
2022-05-18T04:22:04.0885060Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042202.xml
2022-05-18T04:22:05.0109246Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:22:05.0119377Z 
2022-05-18T04:22:05.0119870Z Running tests...
2022-05-18T04:22:05.0120309Z ----------------------------------------------------------------------
2022-05-18T04:22:05.2928330Z   test_barrier_full_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14825
2022-05-18T04:22:05.2950903Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14826
2022-05-18T04:22:05.2974114Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14827
2022-05-18T04:22:06.1633533Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:22:06.1733721Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:22:06.1734184Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:22:06.1734997Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:22:06.1735594Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:22:06.1736116Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:22:06.1843057Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:22:06.1843767Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:22:06.2746895Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:22:06.2955891Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-05-18T04:22:06.2956541Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:22:06.2957068Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:22:06.2957835Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:22:06.2958397Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:22:06.2959023Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:22:07.8044719Z ok (2.792s)
2022-05-18T04:22:07.8044976Z 
2022-05-18T04:22:07.8045490Z ----------------------------------------------------------------------
2022-05-18T04:22:07.8045783Z Ran 1 test in 2.792s
2022-05-18T04:22:07.8045901Z 
2022-05-18T04:22:07.8045970Z OK
2022-05-18T04:22:07.8046049Z 
2022-05-18T04:22:07.8046142Z Generating XML reports...
2022-05-18T04:22:07.8076567Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042205.xml
2022-05-18T04:22:08.7445783Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:22:08.7455714Z 
2022-05-18T04:22:08.7456165Z Running tests...
2022-05-18T04:22:08.7456518Z ----------------------------------------------------------------------
2022-05-18T04:22:09.0291686Z   test_barrier_full_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14887
2022-05-18T04:22:09.0313923Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14888
2022-05-18T04:22:09.0338199Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14889
2022-05-18T04:22:09.8392997Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:22:09.8494251Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:22:09.8494683Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:22:09.8495315Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:22:09.8496142Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:22:09.8497017Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:22:09.8504859Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:22:09.8505389Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:22:09.8506370Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:22:10.0388276Z skip: CUDA is not available. (1.293s)
2022-05-18T04:22:10.0388585Z 
2022-05-18T04:22:10.0389091Z ----------------------------------------------------------------------
2022-05-18T04:22:10.0389701Z Ran 1 test in 1.293s
2022-05-18T04:22:10.0389828Z 
2022-05-18T04:22:10.0389890Z OK (skipped=1)
2022-05-18T04:22:10.0390003Z 
2022-05-18T04:22:10.0390092Z Generating XML reports...
2022-05-18T04:22:10.0420306Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042208.xml
2022-05-18T04:22:10.9580715Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:22:10.9591554Z 
2022-05-18T04:22:10.9592027Z Running tests...
2022-05-18T04:22:10.9592441Z ----------------------------------------------------------------------
2022-05-18T04:22:11.2411090Z   test_barrier_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14940
2022-05-18T04:22:11.2433291Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14941
2022-05-18T04:22:11.2457655Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14942
2022-05-18T04:22:12.0538830Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:22:12.0639290Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:22:12.0640012Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:22:12.0641047Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:22:12.0641949Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:22:12.0642784Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:22:12.0649990Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:22:12.0651487Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:22:12.0652176Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:22:12.0652569Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:22:12.0857271Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-05-18T04:22:12.0857817Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:22:12.0858399Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:22:12.0858916Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:22:12.0955854Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:22:13.1519928Z ok (2.193s)
2022-05-18T04:22:13.1520171Z 
2022-05-18T04:22:13.1520608Z ----------------------------------------------------------------------
2022-05-18T04:22:13.1520886Z Ran 1 test in 2.193s
2022-05-18T04:22:13.1521000Z 
2022-05-18T04:22:13.1521066Z OK
2022-05-18T04:22:13.1521200Z 
2022-05-18T04:22:13.1521313Z Generating XML reports...
2022-05-18T04:22:13.1551459Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042210.xml
2022-05-18T04:22:14.1038091Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:22:14.1047805Z 
2022-05-18T04:22:14.1048083Z Running tests...
2022-05-18T04:22:14.1048557Z ----------------------------------------------------------------------
2022-05-18T04:22:14.3897030Z   test_barrier_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14999
2022-05-18T04:22:14.3919601Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15000
2022-05-18T04:22:14.3943331Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15001
2022-05-18T04:22:15.2073993Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:22:15.2175353Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:22:15.2175940Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:22:15.2176614Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:22:15.2177150Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:22:15.2177695Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:22:15.2184899Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:22:15.2186151Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:22:15.2186704Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:22:15.3995428Z skip: CUDA is not available. (1.294s)
2022-05-18T04:22:15.3995626Z 
2022-05-18T04:22:15.3995916Z ----------------------------------------------------------------------
2022-05-18T04:22:15.3996232Z Ran 1 test in 1.295s
2022-05-18T04:22:15.3996346Z 
2022-05-18T04:22:15.3996418Z OK (skipped=1)
2022-05-18T04:22:15.3996527Z 
2022-05-18T04:22:15.3996613Z Generating XML reports...
2022-05-18T04:22:15.4028355Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042214.xml
2022-05-18T04:22:16.3324483Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:22:16.3334572Z 
2022-05-18T04:22:16.3334668Z Running tests...
2022-05-18T04:22:16.3335165Z ----------------------------------------------------------------------
2022-05-18T04:22:16.6199952Z   test_barrier_timeout_full_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15052
2022-05-18T04:22:16.6221160Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15053
2022-05-18T04:22:16.6245285Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15054
2022-05-18T04:22:17.4192347Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:22:17.4293600Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:22:17.4294238Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:22:17.4295241Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:22:17.4296100Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:22:17.4296734Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:22:17.4401523Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:22:17.5307385Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:22:17.5308170Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:22:17.5514959Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:22:17.5616913Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-05-18T04:22:17.5617465Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:22:17.5618303Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:22:17.5619137Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:22:17.5619715Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:22:18.8313262Z ok (2.498s)
2022-05-18T04:22:18.8313481Z 
2022-05-18T04:22:18.8313812Z ----------------------------------------------------------------------
2022-05-18T04:22:18.8314079Z Ran 1 test in 2.498s
2022-05-18T04:22:18.8314196Z 
2022-05-18T04:22:18.8314258Z OK
2022-05-18T04:22:18.8314349Z 
2022-05-18T04:22:18.8314433Z Generating XML reports...
2022-05-18T04:22:18.8345082Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042216.xml
2022-05-18T04:22:19.7672853Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:22:19.7682705Z 
2022-05-18T04:22:19.7683124Z Running tests...
2022-05-18T04:22:19.7683550Z ----------------------------------------------------------------------
2022-05-18T04:22:19.7703791Z   test_barrier_timeout_global (__main__.TestDistBackendWithSpawn) ... skip: Requires file:// initialization method. Both tcp:// and env:// rely on the TCP store for which reinitialization has proven racy. (0.002s)
2022-05-18T04:22:19.7704140Z 
2022-05-18T04:22:19.7704378Z ----------------------------------------------------------------------
2022-05-18T04:22:19.7704645Z Ran 1 test in 0.002s
2022-05-18T04:22:19.7704749Z 
2022-05-18T04:22:19.7704823Z OK (skipped=1)
2022-05-18T04:22:19.7704932Z 
2022-05-18T04:22:19.7705021Z Generating XML reports...
2022-05-18T04:22:19.7735036Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042219.xml
2022-05-18T04:22:20.5973739Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:22:20.5983500Z 
2022-05-18T04:22:20.5983593Z Running tests...
2022-05-18T04:22:20.5984078Z ----------------------------------------------------------------------
2022-05-18T04:22:20.8801138Z   test_barrier_timeout_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15124
2022-05-18T04:22:20.8823568Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15125
2022-05-18T04:22:20.8847499Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15126
2022-05-18T04:22:21.6973427Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:22:21.6973868Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:22:21.6974236Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:22:21.6975049Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:22:21.6975975Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:22:21.6976838Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:22:21.7080570Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:22:21.7985455Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:22:21.7986104Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:22:21.7988336Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:22:21.8091938Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-05-18T04:22:21.8092424Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:22:21.8093046Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:22:21.8093577Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:22:21.8191628Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:22:26.9964811Z ok (6.398s)
2022-05-18T04:22:26.9965089Z 
2022-05-18T04:22:26.9965566Z ----------------------------------------------------------------------
2022-05-18T04:22:26.9965835Z Ran 1 test in 6.398s
2022-05-18T04:22:26.9965951Z 
2022-05-18T04:22:26.9966001Z OK
2022-05-18T04:22:26.9966092Z 
2022-05-18T04:22:26.9966191Z Generating XML reports...
2022-05-18T04:22:26.9996752Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042220.xml
2022-05-18T04:22:27.9461499Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:22:27.9471610Z 
2022-05-18T04:22:27.9471933Z Running tests...
2022-05-18T04:22:27.9472625Z ----------------------------------------------------------------------
2022-05-18T04:22:28.2300385Z   test_batch_isend_irecv_gloo (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15183
2022-05-18T04:22:28.2322827Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15184
2022-05-18T04:22:28.2346810Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15185
2022-05-18T04:22:29.0072683Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:22:29.0163983Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:22:29.0164425Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:22:29.0165032Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:22:29.0165561Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:22:29.0173442Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:22:29.0272766Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:22:29.0273188Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:22:29.1184071Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:22:29.3399086Z ok (1.392s)
2022-05-18T04:22:29.3399357Z 
2022-05-18T04:22:29.3399845Z ----------------------------------------------------------------------
2022-05-18T04:22:29.3400305Z Ran 1 test in 1.393s
2022-05-18T04:22:29.3400472Z 
2022-05-18T04:22:29.3400534Z OK
2022-05-18T04:22:29.3400629Z 
2022-05-18T04:22:29.3400729Z Generating XML reports...
2022-05-18T04:22:29.3431066Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042227.xml
2022-05-18T04:22:30.2611134Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:22:30.2621248Z 
2022-05-18T04:22:30.2621352Z Running tests...
2022-05-18T04:22:30.2622214Z ----------------------------------------------------------------------
2022-05-18T04:22:30.5442078Z   test_batch_isend_irecv_gloo_tags (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15236
2022-05-18T04:22:30.5462796Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15237
2022-05-18T04:22:30.5487486Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15238
2022-05-18T04:22:31.3182616Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:22:31.3282966Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:22:31.3283425Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:22:31.3284030Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:22:31.3284581Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:22:31.3285098Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:22:31.3390558Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:22:31.4296223Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:22:31.4296647Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:22:31.7540869Z ok (1.492s)
2022-05-18T04:22:31.7541074Z 
2022-05-18T04:22:31.7541519Z ----------------------------------------------------------------------
2022-05-18T04:22:31.7541963Z Ran 1 test in 1.492s
2022-05-18T04:22:31.7542195Z 
2022-05-18T04:22:31.7542291Z OK
2022-05-18T04:22:31.7542402Z 
2022-05-18T04:22:31.7542493Z Generating XML reports...
2022-05-18T04:22:31.7572814Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042230.xml
2022-05-18T04:22:32.6773170Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:22:32.6783555Z 
2022-05-18T04:22:32.6783659Z Running tests...
2022-05-18T04:22:32.6784215Z ----------------------------------------------------------------------
2022-05-18T04:22:32.6805016Z   test_batch_isend_irecv_mixed_backend_err (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2022-05-18T04:22:32.6805329Z 
2022-05-18T04:22:32.6805728Z ----------------------------------------------------------------------
2022-05-18T04:22:32.6806203Z Ran 1 test in 0.002s
2022-05-18T04:22:32.6806382Z 
2022-05-18T04:22:32.6806443Z OK (skipped=1)
2022-05-18T04:22:32.6806552Z 
2022-05-18T04:22:32.6806648Z Generating XML reports...
2022-05-18T04:22:32.6837339Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042232.xml
2022-05-18T04:22:33.5105547Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:22:33.5114671Z 
2022-05-18T04:22:33.5114886Z Running tests...
2022-05-18T04:22:33.5115297Z ----------------------------------------------------------------------
2022-05-18T04:22:33.5137319Z   test_batch_isend_irecv_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2022-05-18T04:22:33.5137875Z 
2022-05-18T04:22:33.5138255Z ----------------------------------------------------------------------
2022-05-18T04:22:33.5138817Z Ran 1 test in 0.002s
2022-05-18T04:22:33.5139017Z 
2022-05-18T04:22:33.5139091Z OK (skipped=1)
2022-05-18T04:22:33.5139201Z 
2022-05-18T04:22:33.5139290Z Generating XML reports...
2022-05-18T04:22:33.5170359Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042233.xml
2022-05-18T04:22:34.3459892Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:22:34.3469882Z 
2022-05-18T04:22:34.3469988Z Running tests...
2022-05-18T04:22:34.3470588Z ----------------------------------------------------------------------
2022-05-18T04:22:34.3492653Z   test_batch_isend_irecv_no_rank_zero_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2022-05-18T04:22:34.3492988Z 
2022-05-18T04:22:34.3493769Z ----------------------------------------------------------------------
2022-05-18T04:22:34.3494074Z Ran 1 test in 0.002s
2022-05-18T04:22:34.3494192Z 
2022-05-18T04:22:34.3494278Z OK (skipped=1)
2022-05-18T04:22:34.3494388Z 
2022-05-18T04:22:34.3494475Z Generating XML reports...
2022-05-18T04:22:34.3525657Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042234.xml
2022-05-18T04:22:35.1844519Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:22:35.1855125Z 
2022-05-18T04:22:35.1855619Z Running tests...
2022-05-18T04:22:35.1856039Z ----------------------------------------------------------------------
2022-05-18T04:22:35.1874125Z   test_batch_isend_irecv_op_err (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2022-05-18T04:22:35.1874409Z 
2022-05-18T04:22:35.1874715Z ----------------------------------------------------------------------
2022-05-18T04:22:35.1874950Z Ran 1 test in 0.002s
2022-05-18T04:22:35.1875064Z 
2022-05-18T04:22:35.1875137Z OK (skipped=1)
2022-05-18T04:22:35.1875245Z 
2022-05-18T04:22:35.1875331Z Generating XML reports...
2022-05-18T04:22:35.1905822Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042235.xml
2022-05-18T04:22:36.0140174Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:22:36.0150428Z 
2022-05-18T04:22:36.0150846Z Running tests...
2022-05-18T04:22:36.0151257Z ----------------------------------------------------------------------
2022-05-18T04:22:36.0168391Z   test_batch_isend_irecv_op_list_err (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2022-05-18T04:22:36.0168623Z 
2022-05-18T04:22:36.0169024Z ----------------------------------------------------------------------
2022-05-18T04:22:36.0169305Z Ran 1 test in 0.002s
2022-05-18T04:22:36.0169425Z 
2022-05-18T04:22:36.0169498Z OK (skipped=1)
2022-05-18T04:22:36.0169637Z 
2022-05-18T04:22:36.0169732Z Generating XML reports...
2022-05-18T04:22:36.0200056Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042236.xml
2022-05-18T04:22:36.8458978Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:22:36.8468839Z 
2022-05-18T04:22:36.8468982Z Running tests...
2022-05-18T04:22:36.8469590Z ----------------------------------------------------------------------
2022-05-18T04:22:36.8492046Z   test_batch_isend_irecv_ring_exchange_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2022-05-18T04:22:36.8492512Z 
2022-05-18T04:22:36.8492814Z ----------------------------------------------------------------------
2022-05-18T04:22:36.8493059Z Ran 1 test in 0.002s
2022-05-18T04:22:36.8493173Z 
2022-05-18T04:22:36.8493244Z OK (skipped=1)
2022-05-18T04:22:36.8493338Z 
2022-05-18T04:22:36.8493421Z Generating XML reports...
2022-05-18T04:22:36.8531828Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042236.xml
2022-05-18T04:22:37.6812775Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:22:37.6823138Z 
2022-05-18T04:22:37.6823237Z Running tests...
2022-05-18T04:22:37.6823851Z ----------------------------------------------------------------------
2022-05-18T04:22:37.6845769Z   test_batch_isend_irecv_self_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2022-05-18T04:22:37.6846189Z 
2022-05-18T04:22:37.6846464Z ----------------------------------------------------------------------
2022-05-18T04:22:37.6846724Z Ran 1 test in 0.002s
2022-05-18T04:22:37.6846836Z 
2022-05-18T04:22:37.6846910Z OK (skipped=1)
2022-05-18T04:22:37.6847004Z 
2022-05-18T04:22:37.6847089Z Generating XML reports...
2022-05-18T04:22:37.6877695Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042237.xml
2022-05-18T04:22:38.5106064Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:22:38.5116903Z 
2022-05-18T04:22:38.5117337Z Running tests...
2022-05-18T04:22:38.5117749Z ----------------------------------------------------------------------
2022-05-18T04:22:38.5134829Z   test_batch_isend_irecv_tensor_err (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2022-05-18T04:22:38.5135234Z 
2022-05-18T04:22:38.5135653Z ----------------------------------------------------------------------
2022-05-18T04:22:38.5136053Z Ran 1 test in 0.002s
2022-05-18T04:22:38.5136236Z 
2022-05-18T04:22:38.5136355Z OK (skipped=1)
2022-05-18T04:22:38.5136516Z 
2022-05-18T04:22:38.5136656Z Generating XML reports...
2022-05-18T04:22:38.5168421Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042238.xml
2022-05-18T04:22:39.3424440Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:22:39.3434619Z 
2022-05-18T04:22:39.3435061Z Running tests...
2022-05-18T04:22:39.3435501Z ----------------------------------------------------------------------
2022-05-18T04:22:39.6241848Z   test_broadcast (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15369
2022-05-18T04:22:39.6263826Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15370
2022-05-18T04:22:39.6286924Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15371
2022-05-18T04:22:40.4531455Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:22:40.4631583Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:22:40.4632209Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:22:40.4632935Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:22:40.4633528Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:22:40.4634041Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:22:40.4739638Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:22:40.5645367Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:22:40.5645988Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:22:40.8338308Z ok (1.490s)
2022-05-18T04:22:40.8338541Z 
2022-05-18T04:22:40.8339077Z ----------------------------------------------------------------------
2022-05-18T04:22:40.8339534Z Ran 1 test in 1.490s
2022-05-18T04:22:40.8339740Z 
2022-05-18T04:22:40.8339851Z OK
2022-05-18T04:22:40.8340018Z 
2022-05-18T04:22:40.8340166Z Generating XML reports...
2022-05-18T04:22:40.8370598Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042239.xml
2022-05-18T04:22:41.7507601Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:22:41.7517306Z 
2022-05-18T04:22:41.7517450Z Running tests...
2022-05-18T04:22:41.7518165Z ----------------------------------------------------------------------
2022-05-18T04:22:42.0339104Z   test_broadcast_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15425
2022-05-18T04:22:42.0360294Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15426
2022-05-18T04:22:42.0384241Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15427
2022-05-18T04:22:42.8356770Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:22:42.8457988Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:22:42.8458644Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:22:42.8459331Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:22:42.8459867Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:22:42.8460384Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:22:42.8468571Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:22:42.8469607Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:22:42.8470827Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:22:43.0434573Z skip: CUDA is not available. (1.291s)
2022-05-18T04:22:43.0434838Z 
2022-05-18T04:22:43.0435281Z ----------------------------------------------------------------------
2022-05-18T04:22:43.0435700Z Ran 1 test in 1.292s
2022-05-18T04:22:43.0435898Z 
2022-05-18T04:22:43.0436008Z OK (skipped=1)
2022-05-18T04:22:43.0436182Z 
2022-05-18T04:22:43.0436296Z Generating XML reports...
2022-05-18T04:22:43.0467796Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042241.xml
2022-05-18T04:22:43.9753276Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:22:43.9763372Z 
2022-05-18T04:22:43.9763499Z Running tests...
2022-05-18T04:22:43.9764118Z ----------------------------------------------------------------------
2022-05-18T04:22:44.2609879Z   test_broadcast_full_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15478
2022-05-18T04:22:44.2631475Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15479
2022-05-18T04:22:44.2654446Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15480
2022-05-18T04:22:45.0766486Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:22:45.0804734Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:22:45.0805198Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:22:45.0805818Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:22:45.0806452Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:22:45.0867723Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:22:45.0914687Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:22:45.0915241Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:22:45.1878663Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:22:45.2026225Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:22:45.2127745Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-05-18T04:22:45.2128454Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:22:45.2129166Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:22:45.2129683Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:22:45.2130203Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:22:45.5708792Z ok (1.594s)
2022-05-18T04:22:45.5709065Z 
2022-05-18T04:22:45.5709732Z ----------------------------------------------------------------------
2022-05-18T04:22:45.5710168Z Ran 1 test in 1.594s
2022-05-18T04:22:45.5710285Z 
2022-05-18T04:22:45.5710347Z OK
2022-05-18T04:22:45.5710438Z 
2022-05-18T04:22:45.5710536Z Generating XML reports...
2022-05-18T04:22:45.5740371Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042243.xml
2022-05-18T04:22:46.4908734Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:22:46.4919088Z 
2022-05-18T04:22:46.4919509Z Running tests...
2022-05-18T04:22:46.4920127Z ----------------------------------------------------------------------
2022-05-18T04:22:46.7759168Z   test_broadcast_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15543
2022-05-18T04:22:46.7780668Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15544
2022-05-18T04:22:46.7803696Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15545
2022-05-18T04:22:47.5951846Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:22:47.5952314Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:22:47.5952685Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:22:47.5953284Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:22:47.5953812Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:22:47.5954333Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:22:47.6058268Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:22:47.6966447Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:22:47.6967111Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:22:47.6969093Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:22:47.7171066Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:22:47.7171716Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-05-18T04:22:47.7172578Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:22:47.7173202Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:22:47.7174049Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:22:48.0856983Z ok (1.593s)
2022-05-18T04:22:48.0857208Z 
2022-05-18T04:22:48.0857663Z ----------------------------------------------------------------------
2022-05-18T04:22:48.0858121Z Ran 1 test in 1.594s
2022-05-18T04:22:48.0858323Z 
2022-05-18T04:22:48.0858405Z OK
2022-05-18T04:22:48.0858485Z 
2022-05-18T04:22:48.0858578Z Generating XML reports...
2022-05-18T04:22:48.0888850Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042246.xml
2022-05-18T04:22:49.0084975Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:22:49.0094415Z 
2022-05-18T04:22:49.0094550Z Running tests...
2022-05-18T04:22:49.0095472Z ----------------------------------------------------------------------
2022-05-18T04:22:49.2925312Z   test_broadcast_multigpu (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15604
2022-05-18T04:22:49.2947315Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15605
2022-05-18T04:22:49.2970982Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15606
2022-05-18T04:22:50.0942460Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:22:50.1042894Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:22:50.1043498Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:22:50.1044509Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:22:50.1045037Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:22:50.1045580Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:22:50.1152570Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:22:50.1153313Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:22:50.2056967Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:22:50.4023240Z skip: CUDA is not available. (1.393s)
2022-05-18T04:22:50.4023546Z 
2022-05-18T04:22:50.4024059Z ----------------------------------------------------------------------
2022-05-18T04:22:50.4024297Z Ran 1 test in 1.393s
2022-05-18T04:22:50.4024411Z 
2022-05-18T04:22:50.4024484Z OK (skipped=1)
2022-05-18T04:22:50.4024594Z 
2022-05-18T04:22:50.4024679Z Generating XML reports...
2022-05-18T04:22:50.4055127Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042249.xml
2022-05-18T04:22:51.3265546Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:22:51.3275558Z 
2022-05-18T04:22:51.3275674Z Running tests...
2022-05-18T04:22:51.3276085Z ----------------------------------------------------------------------
2022-05-18T04:22:51.6114087Z   test_broadcast_object_list (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15657
2022-05-18T04:22:51.6135448Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15658
2022-05-18T04:22:51.6159299Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15659
2022-05-18T04:22:52.4257230Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:22:52.4257973Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:22:52.4258797Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:22:52.4259840Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:22:52.4260701Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:22:52.4261628Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:22:52.4363147Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:22:52.5269796Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:22:52.5270394Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:22:52.7211207Z ok (1.393s)
2022-05-18T04:22:52.7211433Z 
2022-05-18T04:22:52.7211742Z ----------------------------------------------------------------------
2022-05-18T04:22:52.7212015Z Ran 1 test in 1.393s
2022-05-18T04:22:52.7212189Z 
2022-05-18T04:22:52.7212238Z OK
2022-05-18T04:22:52.7212331Z 
2022-05-18T04:22:52.7212429Z Generating XML reports...
2022-05-18T04:22:52.7242883Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042251.xml
2022-05-18T04:22:53.6400604Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:22:53.6410341Z 
2022-05-18T04:22:53.6410574Z Running tests...
2022-05-18T04:22:53.6411179Z ----------------------------------------------------------------------
2022-05-18T04:22:53.6426936Z   test_compute_bucket_assignment_by_size_sparse_error_with_logger (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'nccl', 'gloo'} (0.001s)
2022-05-18T04:22:53.6427464Z 
2022-05-18T04:22:53.6427826Z ----------------------------------------------------------------------
2022-05-18T04:22:53.6428229Z Ran 1 test in 0.002s
2022-05-18T04:22:53.6428432Z 
2022-05-18T04:22:53.6428564Z OK (skipped=1)
2022-05-18T04:22:53.6428751Z 
2022-05-18T04:22:53.6428879Z Generating XML reports...
2022-05-18T04:22:53.6459247Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042253.xml
2022-05-18T04:22:54.4714442Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:22:54.4724151Z 
2022-05-18T04:22:54.4724295Z Running tests...
2022-05-18T04:22:54.4724766Z ----------------------------------------------------------------------
2022-05-18T04:22:54.4739679Z   test_compute_bucket_assignment_by_size_sparse_error_without_logger (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'nccl', 'gloo'} (0.001s)
2022-05-18T04:22:54.4740063Z 
2022-05-18T04:22:54.4740288Z ----------------------------------------------------------------------
2022-05-18T04:22:54.4740530Z Ran 1 test in 0.001s
2022-05-18T04:22:54.4740642Z 
2022-05-18T04:22:54.4740715Z OK (skipped=1)
2022-05-18T04:22:54.4740832Z 
2022-05-18T04:22:54.4740905Z Generating XML reports...
2022-05-18T04:22:54.4772782Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042254.xml
2022-05-18T04:22:55.3059272Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:22:55.3069813Z 
2022-05-18T04:22:55.3069909Z Running tests...
2022-05-18T04:22:55.3070483Z ----------------------------------------------------------------------
2022-05-18T04:22:55.5922719Z   test_ddp_broadcast_buffer (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15730
2022-05-18T04:22:55.5943967Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15731
2022-05-18T04:22:55.5967952Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15732
2022-05-18T04:22:56.3633672Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:22:56.3734027Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:22:56.3734620Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:22:56.3735238Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:22:56.3735754Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:22:56.3736300Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:22:56.3744031Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:22:56.3744640Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:22:56.3745940Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:22:56.6017818Z skip: Need at least 2 CUDA devices (1.294s)
2022-05-18T04:22:56.6018101Z 
2022-05-18T04:22:56.6018617Z ----------------------------------------------------------------------
2022-05-18T04:22:56.6018944Z Ran 1 test in 1.295s
2022-05-18T04:22:56.6019046Z 
2022-05-18T04:22:56.6019122Z OK (skipped=1)
2022-05-18T04:22:56.6019229Z 
2022-05-18T04:22:56.6019315Z Generating XML reports...
2022-05-18T04:22:56.6049928Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042255.xml
2022-05-18T04:22:57.5204457Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:22:57.5213748Z 
2022-05-18T04:22:57.5213862Z Running tests...
2022-05-18T04:22:57.5214333Z ----------------------------------------------------------------------
2022-05-18T04:22:57.8033148Z   test_ddp_broadcast_buffer_via_hook (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15783
2022-05-18T04:22:57.8055635Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15784
2022-05-18T04:22:57.8079716Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15785
2022-05-18T04:22:58.5998310Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:22:58.6097843Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:22:58.6098278Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:22:58.6099066Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:22:58.6099630Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:22:58.6100166Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:22:58.6207876Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:22:58.6208400Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:22:58.7113507Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:22:58.9132557Z skip: Need at least 2 CUDA devices (1.392s)
2022-05-18T04:22:58.9132832Z 
2022-05-18T04:22:58.9133389Z ----------------------------------------------------------------------
2022-05-18T04:22:58.9133668Z Ran 1 test in 1.392s
2022-05-18T04:22:58.9133786Z 
2022-05-18T04:22:58.9134072Z OK (skipped=1)
2022-05-18T04:22:58.9134182Z 
2022-05-18T04:22:58.9134270Z Generating XML reports...
2022-05-18T04:22:58.9165030Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042257.xml
2022-05-18T04:22:59.8355814Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:22:59.8366033Z 
2022-05-18T04:22:59.8366173Z Running tests...
2022-05-18T04:22:59.8366584Z ----------------------------------------------------------------------
2022-05-18T04:23:00.1201317Z   test_ddp_buffer_hook_allreduce (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15836
2022-05-18T04:23:00.1223080Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15837
2022-05-18T04:23:00.1246998Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15838
2022-05-18T04:23:00.9117274Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:23:00.9217621Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:23:00.9218230Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:23:00.9218863Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:23:00.9219390Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:23:00.9219900Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:23:00.9325661Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:23:01.0234651Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:23:01.0248944Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:23:01.2307946Z skip: Need at least 2 CUDA devices (1.394s)
2022-05-18T04:23:01.2308232Z 
2022-05-18T04:23:01.2308548Z ----------------------------------------------------------------------
2022-05-18T04:23:01.2308845Z Ran 1 test in 1.394s
2022-05-18T04:23:01.2308961Z 
2022-05-18T04:23:01.2309035Z OK (skipped=1)
2022-05-18T04:23:01.2309145Z 
2022-05-18T04:23:01.2309232Z Generating XML reports...
2022-05-18T04:23:01.2340221Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042259.xml
2022-05-18T04:23:02.1552875Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:23:02.1563137Z 
2022-05-18T04:23:02.1563272Z Running tests...
2022-05-18T04:23:02.1563724Z ----------------------------------------------------------------------
2022-05-18T04:23:02.4320620Z   test_ddp_buffer_hook_allreduce_return_future (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77261 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure IN_CI is not set and you are not using the flag --import-disabled-tests. (0.275s)
2022-05-18T04:23:02.4321150Z 
2022-05-18T04:23:02.4321363Z ----------------------------------------------------------------------
2022-05-18T04:23:02.4321613Z Ran 1 test in 0.276s
2022-05-18T04:23:02.4321733Z 
2022-05-18T04:23:02.4321807Z OK (skipped=1)
2022-05-18T04:23:02.4321915Z 
2022-05-18T04:23:02.4322002Z Generating XML reports...
2022-05-18T04:23:02.4348190Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042302.xml
2022-05-18T04:23:03.3262372Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:23:03.3272632Z 
2022-05-18T04:23:03.3272731Z Running tests...
2022-05-18T04:23:03.3273151Z ----------------------------------------------------------------------
2022-05-18T04:23:03.3301447Z   test_ddp_build_debug_param_to_name_mapping (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'nccl'} (0.003s)
2022-05-18T04:23:03.3301905Z 
2022-05-18T04:23:03.3302434Z ----------------------------------------------------------------------
2022-05-18T04:23:03.3303058Z Ran 1 test in 0.003s
2022-05-18T04:23:03.3303264Z 
2022-05-18T04:23:03.3303372Z OK (skipped=1)
2022-05-18T04:23:03.3303473Z 
2022-05-18T04:23:03.3303560Z Generating XML reports...
2022-05-18T04:23:03.3333330Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042303.xml
2022-05-18T04:23:04.1614848Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:23:04.1624732Z 
2022-05-18T04:23:04.1624841Z Running tests...
2022-05-18T04:23:04.1625345Z ----------------------------------------------------------------------
2022-05-18T04:23:04.4490925Z   test_ddp_build_debug_param_to_name_mapping_requires_grad (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15909
2022-05-18T04:23:04.4512644Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15910
2022-05-18T04:23:04.4536459Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15911
2022-05-18T04:23:05.2261740Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:23:05.2332882Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:23:05.2333298Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:23:05.2333920Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:23:05.2334476Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:23:05.2363348Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:23:05.2441858Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:23:05.2442489Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:23:05.3376519Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:23:05.4585901Z skip: Need at least 2 CUDA devices (1.296s)
2022-05-18T04:23:05.4586151Z 
2022-05-18T04:23:05.4587079Z ----------------------------------------------------------------------
2022-05-18T04:23:05.4587416Z Ran 1 test in 1.296s
2022-05-18T04:23:05.4587540Z 
2022-05-18T04:23:05.4587647Z OK (skipped=1)
2022-05-18T04:23:05.4587759Z 
2022-05-18T04:23:05.4587833Z Generating XML reports...
2022-05-18T04:23:05.4617398Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042304.xml
2022-05-18T04:23:06.3767218Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:23:06.3777310Z 
2022-05-18T04:23:06.3777406Z Running tests...
2022-05-18T04:23:06.3778394Z ----------------------------------------------------------------------
2022-05-18T04:23:06.6599842Z   test_ddp_comm_hook_logging (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15962
2022-05-18T04:23:06.6621828Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15963
2022-05-18T04:23:06.6644557Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15964
2022-05-18T04:23:07.4901142Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:23:07.5002180Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:23:07.5002945Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:23:07.5003588Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:23:07.5004120Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:23:07.5004643Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:23:07.5109918Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:23:07.6016096Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:23:07.6016682Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:23:07.7695884Z skip: Need at least 3 CUDA devices (1.392s)
2022-05-18T04:23:07.7696087Z 
2022-05-18T04:23:07.7697023Z ----------------------------------------------------------------------
2022-05-18T04:23:07.7697327Z Ran 1 test in 1.392s
2022-05-18T04:23:07.7697448Z 
2022-05-18T04:23:07.7697524Z OK (skipped=1)
2022-05-18T04:23:07.7697631Z 
2022-05-18T04:23:07.7697703Z Generating XML reports...
2022-05-18T04:23:07.7728183Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042306.xml
2022-05-18T04:23:08.7074478Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:23:08.7085064Z 
2022-05-18T04:23:08.7085545Z Running tests...
2022-05-18T04:23:08.7085964Z ----------------------------------------------------------------------
2022-05-18T04:23:08.7124593Z   test_ddp_control_flow_different_across_ranks (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'nccl'} (0.004s)
2022-05-18T04:23:08.7125081Z 
2022-05-18T04:23:08.7125461Z ----------------------------------------------------------------------
2022-05-18T04:23:08.7125781Z Ran 1 test in 0.004s
2022-05-18T04:23:08.7125895Z 
2022-05-18T04:23:08.7125969Z OK (skipped=1)
2022-05-18T04:23:08.7126076Z 
2022-05-18T04:23:08.7126162Z Generating XML reports...
2022-05-18T04:23:08.7157216Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042308.xml
2022-05-18T04:23:09.5454577Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:23:09.5464776Z 
2022-05-18T04:23:09.5464917Z Running tests...
2022-05-18T04:23:09.5465358Z ----------------------------------------------------------------------
2022-05-18T04:23:09.5498443Z   test_ddp_control_flow_same_across_ranks (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'nccl'} (0.003s)
2022-05-18T04:23:09.5498874Z 
2022-05-18T04:23:09.5499141Z ----------------------------------------------------------------------
2022-05-18T04:23:09.5499456Z Ran 1 test in 0.003s
2022-05-18T04:23:09.5499570Z 
2022-05-18T04:23:09.5499650Z OK (skipped=1)
2022-05-18T04:23:09.5499756Z 
2022-05-18T04:23:09.5499832Z Generating XML reports...
2022-05-18T04:23:09.5531261Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042309.xml
2022-05-18T04:23:10.3821531Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:23:10.3831558Z 
2022-05-18T04:23:10.3831667Z Running tests...
2022-05-18T04:23:10.3832090Z ----------------------------------------------------------------------
2022-05-18T04:23:10.6664478Z   test_ddp_create_graph (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16035
2022-05-18T04:23:10.6685871Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16036
2022-05-18T04:23:10.6710089Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16037
2022-05-18T04:23:11.4937629Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:23:11.5039108Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:23:11.5040271Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:23:11.5040833Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:23:11.5041783Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:23:11.5042343Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:23:11.5048434Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:23:11.5049585Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:23:11.5050414Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:23:11.5113890Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkn8ozu08
2022-05-18T04:23:11.5114791Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpoer1rby2
2022-05-18T04:23:11.5115507Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpud75h_mh
2022-05-18T04:23:11.5116224Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkn8ozu08/_remote_module_non_scriptable.py
2022-05-18T04:23:11.5118266Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpoer1rby2/_remote_module_non_scriptable.py
2022-05-18T04:23:11.5118939Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpud75h_mh/_remote_module_non_scriptable.py
2022-05-18T04:23:11.5223426Z [W reducer.cpp:367] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-05-18T04:23:11.5224628Z [W reducer.cpp:367] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-05-18T04:23:11.5225916Z [W reducer.cpp:367] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-05-18T04:23:11.5229864Z /opt/conda/lib/python3.7/site-packages/torch/autograd/__init__.py:175: UserWarning: Using backward() with create_graph=True will create a reference cycle between the parameter and its gradient which can cause a memory leak. We recommend using autograd.grad when creating the graph to avoid this. If you have to use this function, make sure to reset the .grad fields of your parameters to None after use to break the cycle and avoid the leak. (Triggered internally at  /var/lib/jenkins/workspace/torch/csrc/autograd/engine.cpp:995.)
2022-05-18T04:23:11.5230746Z   allow_unreachable=True, accumulate_grad=True)  # Calls into the C++ engine to run the backward pass
2022-05-18T04:23:11.5232578Z /opt/conda/lib/python3.7/site-packages/torch/autograd/__init__.py:175: UserWarning: Using backward() with create_graph=True will create a reference cycle between the parameter and its gradient which can cause a memory leak. We recommend using autograd.grad when creating the graph to avoid this. If you have to use this function, make sure to reset the .grad fields of your parameters to None after use to break the cycle and avoid the leak. (Triggered internally at  /var/lib/jenkins/workspace/torch/csrc/autograd/engine.cpp:995.)
2022-05-18T04:23:11.5233656Z   allow_unreachable=True, accumulate_grad=True)  # Calls into the C++ engine to run the backward pass
2022-05-18T04:23:11.5235380Z /opt/conda/lib/python3.7/site-packages/torch/autograd/__init__.py:175: UserWarning: Using backward() with create_graph=True will create a reference cycle between the parameter and its gradient which can cause a memory leak. We recommend using autograd.grad when creating the graph to avoid this. If you have to use this function, make sure to reset the .grad fields of your parameters to None after use to break the cycle and avoid the leak. (Triggered internally at  /var/lib/jenkins/workspace/torch/csrc/autograd/engine.cpp:995.)
2022-05-18T04:23:11.5236123Z   allow_unreachable=True, accumulate_grad=True)  # Calls into the C++ engine to run the backward pass
2022-05-18T04:23:11.5236481Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:23:11.5236840Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:23:11.5237195Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:23:11.5238073Z [W reducer.cpp:367] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-05-18T04:23:11.5239448Z [W reducer.cpp:367] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-05-18T04:23:11.5241158Z [W reducer.cpp:367] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-05-18T04:23:11.5242411Z [W reducer.cpp:367] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-05-18T04:23:11.5243536Z [W reducer.cpp:367] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-05-18T04:23:11.5244785Z [W reducer.cpp:367] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-05-18T04:23:11.5247080Z [W reducer.cpp:367] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-05-18T04:23:11.5248693Z [W reducer.cpp:367] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-05-18T04:23:11.5249885Z [W reducer.cpp:367] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-05-18T04:23:11.5252001Z [W reducer.cpp:367] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-05-18T04:23:11.5253325Z [W reducer.cpp:367] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-05-18T04:23:11.5254443Z [W reducer.cpp:367] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-05-18T04:23:11.5257616Z [W reducer.cpp:367] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-05-18T04:23:11.5258907Z [W reducer.cpp:367] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-05-18T04:23:11.5260051Z [W reducer.cpp:367] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-05-18T04:23:11.6761104Z ok (1.293s)
2022-05-18T04:23:11.6761368Z 
2022-05-18T04:23:11.6763209Z ----------------------------------------------------------------------
2022-05-18T04:23:11.6763516Z Ran 1 test in 1.293s
2022-05-18T04:23:11.6763647Z 
2022-05-18T04:23:11.6763708Z OK
2022-05-18T04:23:11.6763800Z 
2022-05-18T04:23:11.6763895Z Generating XML reports...
2022-05-18T04:23:11.6793239Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042310.xml
2022-05-18T04:23:12.6025685Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:23:12.6035102Z 
2022-05-18T04:23:12.6035244Z Running tests...
2022-05-18T04:23:12.6036572Z ----------------------------------------------------------------------
2022-05-18T04:23:12.6077631Z   test_ddp_device (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'nccl'} (0.004s)
2022-05-18T04:23:12.6078058Z 
2022-05-18T04:23:12.6078451Z ----------------------------------------------------------------------
2022-05-18T04:23:12.6078861Z Ran 1 test in 0.004s
2022-05-18T04:23:12.6078964Z 
2022-05-18T04:23:12.6079038Z OK (skipped=1)
2022-05-18T04:23:12.6079153Z 
2022-05-18T04:23:12.6079240Z Generating XML reports...
2022-05-18T04:23:12.6109614Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042312.xml
2022-05-18T04:23:13.4408011Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:23:13.4418203Z 
2022-05-18T04:23:13.4418314Z Running tests...
2022-05-18T04:23:13.4418893Z ----------------------------------------------------------------------
2022-05-18T04:23:13.4445817Z   test_ddp_forward_backward_hook (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'nccl', 'gloo'} (0.003s)
2022-05-18T04:23:13.4446259Z 
2022-05-18T04:23:13.4446661Z ----------------------------------------------------------------------
2022-05-18T04:23:13.4447004Z Ran 1 test in 0.003s
2022-05-18T04:23:13.4447117Z 
2022-05-18T04:23:13.4447191Z OK (skipped=1)
2022-05-18T04:23:13.4447301Z 
2022-05-18T04:23:13.4447376Z Generating XML reports...
2022-05-18T04:23:13.4477851Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042313.xml
2022-05-18T04:23:14.2772093Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:23:14.2782458Z 
2022-05-18T04:23:14.2782592Z Running tests...
2022-05-18T04:23:14.2783126Z ----------------------------------------------------------------------
2022-05-18T04:23:14.5595868Z   test_ddp_grad_div_uneven_inputs (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16108
2022-05-18T04:23:14.5616727Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16109
2022-05-18T04:23:14.5640020Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16110
2022-05-18T04:23:15.3404914Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:23:15.3504633Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:23:15.3505125Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:23:15.3505955Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:23:15.3506656Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:23:15.3507240Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:23:15.3612959Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:23:15.4523039Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:23:15.4523457Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:23:15.6691698Z skip: Need at least 2 CUDA devices (1.391s)
2022-05-18T04:23:15.6691993Z 
2022-05-18T04:23:15.6692518Z ----------------------------------------------------------------------
2022-05-18T04:23:15.6692939Z Ran 1 test in 1.391s
2022-05-18T04:23:15.6693041Z 
2022-05-18T04:23:15.6693351Z OK (skipped=1)
2022-05-18T04:23:15.6693462Z 
2022-05-18T04:23:15.6693551Z Generating XML reports...
2022-05-18T04:23:15.6723611Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042314.xml
2022-05-18T04:23:16.5912997Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:23:16.5922875Z 
2022-05-18T04:23:16.5923414Z Running tests...
2022-05-18T04:23:16.5923814Z ----------------------------------------------------------------------
2022-05-18T04:23:16.8649129Z   test_ddp_hook_parity_allreduce (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77293 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure IN_CI is not set and you are not using the flag --import-disabled-tests. (0.272s)
2022-05-18T04:23:16.8649638Z 
2022-05-18T04:23:16.8649861Z ----------------------------------------------------------------------
2022-05-18T04:23:16.8650122Z Ran 1 test in 0.272s
2022-05-18T04:23:16.8650227Z 
2022-05-18T04:23:16.8650300Z OK (skipped=1)
2022-05-18T04:23:16.8650416Z 
2022-05-18T04:23:16.8650502Z Generating XML reports...
2022-05-18T04:23:16.8678729Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042316.xml
2022-05-18T04:23:17.7627217Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:23:17.7637290Z 
2022-05-18T04:23:17.7637386Z Running tests...
2022-05-18T04:23:17.7639100Z ----------------------------------------------------------------------
2022-05-18T04:23:18.0438359Z   test_ddp_hook_parity_allreduce_process_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16171
2022-05-18T04:23:18.0460295Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16172
2022-05-18T04:23:18.0483413Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16173
2022-05-18T04:23:18.8502198Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:23:18.8602886Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:23:18.8603533Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:23:18.8604506Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:23:18.8605372Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:23:18.8605910Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:23:18.8710222Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:23:18.9617229Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:23:18.9617970Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:23:19.1535125Z skip: Need at least 3 CUDA devices (1.389s)
2022-05-18T04:23:19.1535368Z 
2022-05-18T04:23:19.1535805Z ----------------------------------------------------------------------
2022-05-18T04:23:19.1536210Z Ran 1 test in 1.390s
2022-05-18T04:23:19.1536387Z 
2022-05-18T04:23:19.1536501Z OK (skipped=1)
2022-05-18T04:23:19.1536664Z 
2022-05-18T04:23:19.1536801Z Generating XML reports...
2022-05-18T04:23:19.1568018Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042317.xml
2022-05-18T04:23:20.0820818Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:23:20.0831078Z 
2022-05-18T04:23:20.0831206Z Running tests...
2022-05-18T04:23:20.0831802Z ----------------------------------------------------------------------
2022-05-18T04:23:20.3649212Z   test_ddp_hook_parity_post_localSGD (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16224
2022-05-18T04:23:20.3671659Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16225
2022-05-18T04:23:20.3695165Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16226
2022-05-18T04:23:21.1670624Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:23:21.1769476Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:23:21.1770213Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:23:21.1770855Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:23:21.1771631Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:23:21.1772351Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:23:21.1878600Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:23:21.1879240Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:23:21.2786006Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:23:21.4747389Z skip: Need at least 3 CUDA devices (1.391s)
2022-05-18T04:23:21.4747661Z 
2022-05-18T04:23:21.4748115Z ----------------------------------------------------------------------
2022-05-18T04:23:21.4748494Z Ran 1 test in 1.392s
2022-05-18T04:23:21.4748695Z 
2022-05-18T04:23:21.4748815Z OK (skipped=1)
2022-05-18T04:23:21.4748979Z 
2022-05-18T04:23:21.4749114Z Generating XML reports...
2022-05-18T04:23:21.4780181Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042320.xml
2022-05-18T04:23:22.3987219Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:23:22.3997735Z 
2022-05-18T04:23:22.3998058Z Running tests...
2022-05-18T04:23:22.3998782Z ----------------------------------------------------------------------
2022-05-18T04:23:22.6832518Z   test_ddp_hook_parity_powerSGD (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16277
2022-05-18T04:23:22.6854172Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16278
2022-05-18T04:23:22.6877160Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16279
2022-05-18T04:23:23.4868541Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:23:23.4963005Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:23:23.4963425Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:23:23.4964033Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:23:23.4964570Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:23:23.4969361Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:23:23.5073008Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:23:23.5073578Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:23:23.5983162Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:23:23.7927567Z skip: Need at least 3 CUDA devices (1.393s)
2022-05-18T04:23:23.7928128Z 
2022-05-18T04:23:23.7928650Z ----------------------------------------------------------------------
2022-05-18T04:23:23.7929118Z Ran 1 test in 1.393s
2022-05-18T04:23:23.7929332Z 
2022-05-18T04:23:23.7929440Z OK (skipped=1)
2022-05-18T04:23:23.7929567Z 
2022-05-18T04:23:23.7929655Z Generating XML reports...
2022-05-18T04:23:23.7959553Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042322.xml
2022-05-18T04:23:24.7171746Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:23:24.7181409Z 
2022-05-18T04:23:24.7181560Z Running tests...
2022-05-18T04:23:24.7182303Z ----------------------------------------------------------------------
2022-05-18T04:23:25.0003906Z   test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16330
2022-05-18T04:23:25.0026045Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16331
2022-05-18T04:23:25.0048952Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16332
2022-05-18T04:23:25.8250416Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:23:25.8351187Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:23:25.8351684Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:23:25.8352291Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:23:25.8352820Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:23:25.8353365Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:23:25.8458307Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:23:25.9365430Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:23:25.9367186Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:23:26.1099728Z skip: Need at least 2 CUDA devices (1.391s)
2022-05-18T04:23:26.1099995Z 
2022-05-18T04:23:26.1100346Z ----------------------------------------------------------------------
2022-05-18T04:23:26.1100630Z Ran 1 test in 1.392s
2022-05-18T04:23:26.1100745Z 
2022-05-18T04:23:26.1100821Z OK (skipped=1)
2022-05-18T04:23:26.1100981Z 
2022-05-18T04:23:26.1101148Z Generating XML reports...
2022-05-18T04:23:26.1132021Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042324.xml
2022-05-18T04:23:27.0277708Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:23:27.0287223Z 
2022-05-18T04:23:27.0287323Z Running tests...
2022-05-18T04:23:27.0287881Z ----------------------------------------------------------------------
2022-05-18T04:23:27.3089111Z   test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16383
2022-05-18T04:23:27.3111696Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16384
2022-05-18T04:23:27.3135175Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16385
2022-05-18T04:23:28.1234046Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:23:28.1252072Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:23:28.1252654Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:23:28.1253291Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:23:28.1253852Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:23:28.1334808Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:23:28.1360669Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:23:28.1361353Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:23:28.2347461Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:23:28.4184938Z skip: Need at least 2 CUDA devices (1.389s)
2022-05-18T04:23:28.4185166Z 
2022-05-18T04:23:28.4186347Z ----------------------------------------------------------------------
2022-05-18T04:23:28.4186636Z Ran 1 test in 1.390s
2022-05-18T04:23:28.4186753Z 
2022-05-18T04:23:28.4186825Z OK (skipped=1)
2022-05-18T04:23:28.4186936Z 
2022-05-18T04:23:28.4187024Z Generating XML reports...
2022-05-18T04:23:28.4217434Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042327.xml
2022-05-18T04:23:29.3438026Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:23:29.3447563Z 
2022-05-18T04:23:29.3447656Z Running tests...
2022-05-18T04:23:29.3448080Z ----------------------------------------------------------------------
2022-05-18T04:23:29.6274533Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16436
2022-05-18T04:23:29.6297138Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16437
2022-05-18T04:23:29.6319603Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16438
2022-05-18T04:23:30.4041327Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:23:30.4041830Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:23:30.4042250Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:23:30.4042853Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:23:30.4043377Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:23:30.4141652Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:23:30.4151715Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:23:30.4152313Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:23:30.4152871Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:23:30.6368732Z skip: Need at least 2 CUDA devices (1.292s)
2022-05-18T04:23:30.6369025Z 
2022-05-18T04:23:30.6369556Z ----------------------------------------------------------------------
2022-05-18T04:23:30.6369884Z Ran 1 test in 1.292s
2022-05-18T04:23:30.6369985Z 
2022-05-18T04:23:30.6370059Z OK (skipped=1)
2022-05-18T04:23:30.6370166Z 
2022-05-18T04:23:30.6370250Z Generating XML reports...
2022-05-18T04:23:30.6400215Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042329.xml
2022-05-18T04:23:31.5580782Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:23:31.5591669Z 
2022-05-18T04:23:31.5591798Z Running tests...
2022-05-18T04:23:31.5592184Z ----------------------------------------------------------------------
2022-05-18T04:23:31.8404859Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16489
2022-05-18T04:23:31.8426107Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16490
2022-05-18T04:23:31.8448722Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16491
2022-05-18T04:23:32.6853108Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:23:32.6953955Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:23:32.6954724Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:23:32.6955404Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:23:32.6955948Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:23:32.6956457Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:23:32.6964829Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:23:32.6965776Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:23:32.6967442Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:23:32.9498203Z skip: Need at least 2 CUDA devices (1.390s)
2022-05-18T04:23:32.9498653Z 
2022-05-18T04:23:32.9499545Z ----------------------------------------------------------------------
2022-05-18T04:23:32.9500063Z Ran 1 test in 1.391s
2022-05-18T04:23:32.9500184Z 
2022-05-18T04:23:32.9500265Z OK (skipped=1)
2022-05-18T04:23:32.9500377Z 
2022-05-18T04:23:32.9500473Z Generating XML reports...
2022-05-18T04:23:32.9530810Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042331.xml
2022-05-18T04:23:33.8738868Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:23:33.8749674Z 
2022-05-18T04:23:33.8749968Z Running tests...
2022-05-18T04:23:33.8750664Z ----------------------------------------------------------------------
2022-05-18T04:23:34.1571604Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16542
2022-05-18T04:23:34.1594034Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16543
2022-05-18T04:23:34.1616629Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16544
2022-05-18T04:23:34.9839607Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:23:34.9908779Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:23:34.9909437Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:23:34.9910067Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:23:34.9910593Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:23:34.9939628Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:23:35.0018002Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:23:35.0018830Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:23:35.0952179Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:23:35.2667386Z skip: Need at least 2 CUDA devices (1.391s)
2022-05-18T04:23:35.2667632Z 
2022-05-18T04:23:35.2668036Z ----------------------------------------------------------------------
2022-05-18T04:23:35.2668293Z Ran 1 test in 1.392s
2022-05-18T04:23:35.2668396Z 
2022-05-18T04:23:35.2668472Z OK (skipped=1)
2022-05-18T04:23:35.2668581Z 
2022-05-18T04:23:35.2668668Z Generating XML reports...
2022-05-18T04:23:35.2700112Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042333.xml
2022-05-18T04:23:36.1874016Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:23:36.1884447Z 
2022-05-18T04:23:36.1884597Z Running tests...
2022-05-18T04:23:36.1885191Z ----------------------------------------------------------------------
2022-05-18T04:23:36.4701025Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16595
2022-05-18T04:23:36.4723461Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16596
2022-05-18T04:23:36.4746366Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16597
2022-05-18T04:23:37.3056304Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:23:37.3056951Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:23:37.3057393Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:23:37.3058074Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:23:37.3058937Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:23:37.3059774Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:23:37.3164651Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:23:37.4071511Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:23:37.4072169Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:23:37.5797578Z skip: Need at least 2 CUDA devices (1.391s)
2022-05-18T04:23:37.5797793Z 
2022-05-18T04:23:37.5798129Z ----------------------------------------------------------------------
2022-05-18T04:23:37.5798429Z Ran 1 test in 1.391s
2022-05-18T04:23:37.5798558Z 
2022-05-18T04:23:37.5798676Z OK (skipped=1)
2022-05-18T04:23:37.5798788Z 
2022-05-18T04:23:37.5798876Z Generating XML reports...
2022-05-18T04:23:37.5829426Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042336.xml
2022-05-18T04:23:38.5071950Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:23:38.5081489Z 
2022-05-18T04:23:38.5081652Z Running tests...
2022-05-18T04:23:38.5082270Z ----------------------------------------------------------------------
2022-05-18T04:23:38.7909441Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16648
2022-05-18T04:23:38.7931326Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16649
2022-05-18T04:23:38.7954435Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16650
2022-05-18T04:23:39.6232734Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:23:39.6333621Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:23:39.6334146Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:23:39.6335296Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:23:39.6335831Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:23:39.6336373Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:23:39.6441705Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:23:39.7350637Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:23:39.7351336Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:23:39.9005729Z skip: Need at least 2 CUDA devices (1.392s)
2022-05-18T04:23:39.9006021Z 
2022-05-18T04:23:39.9006549Z ----------------------------------------------------------------------
2022-05-18T04:23:39.9006808Z Ran 1 test in 1.392s
2022-05-18T04:23:39.9006921Z 
2022-05-18T04:23:39.9006997Z OK (skipped=1)
2022-05-18T04:23:39.9007105Z 
2022-05-18T04:23:39.9007192Z Generating XML reports...
2022-05-18T04:23:39.9038000Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042338.xml
2022-05-18T04:23:40.8231014Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:23:40.8240896Z 
2022-05-18T04:23:40.8241027Z Running tests...
2022-05-18T04:23:40.8241615Z ----------------------------------------------------------------------
2022-05-18T04:23:41.1046357Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16701
2022-05-18T04:23:41.1069248Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16702
2022-05-18T04:23:41.1092364Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16703
2022-05-18T04:23:41.9127967Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:23:41.9128561Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:23:41.9128966Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:23:41.9129583Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:23:41.9130112Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:23:41.9130636Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:23:41.9233528Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:23:42.0142386Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:23:42.0142818Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:23:42.2142271Z skip: Need at least 2 CUDA devices (1.390s)
2022-05-18T04:23:42.2143095Z 
2022-05-18T04:23:42.2143666Z ----------------------------------------------------------------------
2022-05-18T04:23:42.2143977Z Ran 1 test in 1.390s
2022-05-18T04:23:42.2144181Z 
2022-05-18T04:23:42.2144243Z OK (skipped=1)
2022-05-18T04:23:42.2144351Z 
2022-05-18T04:23:42.2144437Z Generating XML reports...
2022-05-18T04:23:42.2174593Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042340.xml
2022-05-18T04:23:43.1481306Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:23:43.1491110Z 
2022-05-18T04:23:43.1491325Z Running tests...
2022-05-18T04:23:43.1491778Z ----------------------------------------------------------------------
2022-05-18T04:23:43.4357993Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16754
2022-05-18T04:23:43.4379794Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16755
2022-05-18T04:23:43.4402942Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16756
2022-05-18T04:23:44.2458670Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:23:44.2459310Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:23:44.2459898Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:23:44.2460894Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:23:44.2461555Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:23:44.2462084Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:23:44.2566919Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:23:44.2567669Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:23:44.3471362Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:23:44.5453773Z skip: Need at least 2 CUDA devices (1.396s)
2022-05-18T04:23:44.5454059Z 
2022-05-18T04:23:44.5454432Z ----------------------------------------------------------------------
2022-05-18T04:23:44.5454704Z Ran 1 test in 1.396s
2022-05-18T04:23:44.5454818Z 
2022-05-18T04:23:44.5454891Z OK (skipped=1)
2022-05-18T04:23:44.5454999Z 
2022-05-18T04:23:44.5455074Z Generating XML reports...
2022-05-18T04:23:44.5486827Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042343.xml
2022-05-18T04:23:45.4654277Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:23:45.4663871Z 
2022-05-18T04:23:45.4664021Z Running tests...
2022-05-18T04:23:45.4664671Z ----------------------------------------------------------------------
2022-05-18T04:23:45.7554743Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16807
2022-05-18T04:23:45.7577205Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16808
2022-05-18T04:23:45.7600705Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16809
2022-05-18T04:23:46.5617977Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:23:46.5718764Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:23:46.5719675Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:23:46.5720380Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:23:46.5720909Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:23:46.5721415Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:23:46.5826247Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:23:46.6733671Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:23:46.6734198Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:23:46.8653197Z skip: Need at least 2 CUDA devices (1.399s)
2022-05-18T04:23:46.8653396Z 
2022-05-18T04:23:46.8653854Z ----------------------------------------------------------------------
2022-05-18T04:23:46.8654109Z Ran 1 test in 1.399s
2022-05-18T04:23:46.8654211Z 
2022-05-18T04:23:46.8654291Z OK (skipped=1)
2022-05-18T04:23:46.8654402Z 
2022-05-18T04:23:46.8654488Z Generating XML reports...
2022-05-18T04:23:46.8685792Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042345.xml
2022-05-18T04:23:47.8144345Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:23:47.8154296Z 
2022-05-18T04:23:47.8154409Z Running tests...
2022-05-18T04:23:47.8154980Z ----------------------------------------------------------------------
2022-05-18T04:23:48.1010820Z   test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16860
2022-05-18T04:23:48.1033324Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16861
2022-05-18T04:23:48.1056017Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16862
2022-05-18T04:23:48.9307321Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:23:48.9408561Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:23:48.9409026Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:23:48.9409658Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:23:48.9410176Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:23:48.9410699Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:23:48.9516443Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:23:49.0423622Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:23:49.0424156Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:23:49.2105123Z skip: Need at least 2 CUDA devices (1.395s)
2022-05-18T04:23:49.2105464Z 
2022-05-18T04:23:49.2105896Z ----------------------------------------------------------------------
2022-05-18T04:23:49.2106339Z Ran 1 test in 1.395s
2022-05-18T04:23:49.2106535Z 
2022-05-18T04:23:49.2106625Z OK (skipped=1)
2022-05-18T04:23:49.2106733Z 
2022-05-18T04:23:49.2106820Z Generating XML reports...
2022-05-18T04:23:49.2137155Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042347.xml
2022-05-18T04:23:50.1401557Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:23:50.1411959Z 
2022-05-18T04:23:50.1412465Z Running tests...
2022-05-18T04:23:50.1412881Z ----------------------------------------------------------------------
2022-05-18T04:23:50.4217246Z   test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16913
2022-05-18T04:23:50.4239807Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16914
2022-05-18T04:23:50.4263538Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16915
2022-05-18T04:23:51.2381436Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:23:51.2481940Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:23:51.2482684Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:23:51.2483515Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:23:51.2484058Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:23:51.2484568Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:23:51.2492235Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:23:51.2492801Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:23:51.2493957Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:23:51.4312672Z skip: Need at least 2 CUDA devices (1.290s)
2022-05-18T04:23:51.4313023Z 
2022-05-18T04:23:51.4313536Z ----------------------------------------------------------------------
2022-05-18T04:23:51.4313878Z Ran 1 test in 1.290s
2022-05-18T04:23:51.4313990Z 
2022-05-18T04:23:51.4314064Z OK (skipped=1)
2022-05-18T04:23:51.4314160Z 
2022-05-18T04:23:51.4314246Z Generating XML reports...
2022-05-18T04:23:51.4344896Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042350.xml
2022-05-18T04:23:52.3551152Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:23:52.3561156Z 
2022-05-18T04:23:52.3561383Z Running tests...
2022-05-18T04:23:52.3561977Z ----------------------------------------------------------------------
2022-05-18T04:23:52.3577896Z   test_ddp_ignore_params_arg (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'nccl', 'gloo'} (0.002s)
2022-05-18T04:23:52.3578335Z 
2022-05-18T04:23:52.3579087Z ----------------------------------------------------------------------
2022-05-18T04:23:52.3579452Z Ran 1 test in 0.002s
2022-05-18T04:23:52.3579592Z 
2022-05-18T04:23:52.3579668Z OK (skipped=1)
2022-05-18T04:23:52.3579777Z 
2022-05-18T04:23:52.3579864Z Generating XML reports...
2022-05-18T04:23:52.3611168Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042352.xml
2022-05-18T04:23:53.2336270Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:23:53.2347354Z 
2022-05-18T04:23:53.2347686Z Running tests...
2022-05-18T04:23:53.2348343Z ----------------------------------------------------------------------
2022-05-18T04:23:53.5175433Z   test_ddp_inference (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16976
2022-05-18T04:23:53.5198492Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16977
2022-05-18T04:23:53.5221800Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16978
2022-05-18T04:23:54.3183568Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:23:54.3284733Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:23:54.3285695Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:23:54.3286103Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:23:54.3286596Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:23:54.3287118Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:23:54.3294599Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:23:54.3296924Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:23:54.3297453Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:23:54.5273131Z skip: Need at least 2 CUDA devices (1.292s)
2022-05-18T04:23:54.5273325Z 
2022-05-18T04:23:54.5273683Z ----------------------------------------------------------------------
2022-05-18T04:23:54.5274086Z Ran 1 test in 1.293s
2022-05-18T04:23:54.5274299Z 
2022-05-18T04:23:54.5274433Z OK (skipped=1)
2022-05-18T04:23:54.5274616Z 
2022-05-18T04:23:54.5274770Z Generating XML reports...
2022-05-18T04:23:54.5305107Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042353.xml
2022-05-18T04:23:55.5258051Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:23:55.5258536Z 
2022-05-18T04:23:55.5258664Z Running tests...
2022-05-18T04:23:55.5259300Z ----------------------------------------------------------------------
2022-05-18T04:23:55.8373564Z   test_ddp_join_model_equivalence (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17029
2022-05-18T04:23:55.8395773Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17030
2022-05-18T04:23:55.8418529Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17031
2022-05-18T04:23:56.6433990Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:23:56.6534368Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:23:56.6535359Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:23:56.6536024Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:23:56.6536788Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:23:56.6537707Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:23:56.6641957Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:23:56.7549642Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:23:56.7550308Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:23:56.9469261Z skip: Need at least 2 CUDA devices (1.421s)
2022-05-18T04:23:56.9469565Z 
2022-05-18T04:23:56.9470031Z ----------------------------------------------------------------------
2022-05-18T04:23:56.9470424Z Ran 1 test in 1.421s
2022-05-18T04:23:56.9470608Z 
2022-05-18T04:23:56.9470730Z OK (skipped=1)
2022-05-18T04:23:56.9470904Z 
2022-05-18T04:23:56.9471048Z Generating XML reports...
2022-05-18T04:23:56.9502157Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042355.xml
2022-05-18T04:23:57.8770985Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:23:57.8780868Z 
2022-05-18T04:23:57.8780997Z Running tests...
2022-05-18T04:23:57.8781432Z ----------------------------------------------------------------------
2022-05-18T04:23:58.1786128Z   test_ddp_logging_data_cpu (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17082
2022-05-18T04:23:58.1807760Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17083
2022-05-18T04:23:58.1830845Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17084
2022-05-18T04:23:59.0289879Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:23:59.0390828Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:23:59.0391427Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:23:59.0392394Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:23:59.0393246Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:23:59.0394112Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:23:59.0499607Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:23:59.0500006Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:23:59.0565579Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp28w0icn6
2022-05-18T04:23:59.0567552Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqur9gt4e
2022-05-18T04:23:59.0567951Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp28w0icn6/_remote_module_non_scriptable.py
2022-05-18T04:23:59.0569821Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqur9gt4e/_remote_module_non_scriptable.py
2022-05-18T04:23:59.1405043Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:23:59.1472151Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu4pkb89p
2022-05-18T04:23:59.1473961Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu4pkb89p/_remote_module_non_scriptable.py
2022-05-18T04:23:59.1604115Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:23:59.1604818Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:23:59.1605459Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:23:59.4883905Z ok (1.610s)
2022-05-18T04:23:59.4884144Z 
2022-05-18T04:23:59.4884470Z ----------------------------------------------------------------------
2022-05-18T04:23:59.4884743Z Ran 1 test in 1.610s
2022-05-18T04:23:59.4884860Z 
2022-05-18T04:23:59.4884926Z OK
2022-05-18T04:23:59.4885006Z 
2022-05-18T04:23:59.4885099Z Generating XML reports...
2022-05-18T04:23:59.4917583Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042357.xml
2022-05-18T04:24:00.4123242Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:24:00.4133230Z 
2022-05-18T04:24:00.4133327Z Running tests...
2022-05-18T04:24:00.4134335Z ----------------------------------------------------------------------
2022-05-18T04:24:00.6943351Z   test_ddp_logging_data_gpu (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17144
2022-05-18T04:24:00.6964596Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17145
2022-05-18T04:24:00.6987342Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17146
2022-05-18T04:24:01.5197690Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:24:01.5299737Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:24:01.5300416Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:24:01.5301349Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:24:01.5302166Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:24:01.5303076Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:24:01.5311876Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:24:01.5313584Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:24:01.5313946Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:24:01.8038312Z skip: CUDA is not available. (1.390s)
2022-05-18T04:24:01.8038628Z 
2022-05-18T04:24:01.8039155Z ----------------------------------------------------------------------
2022-05-18T04:24:01.8039454Z Ran 1 test in 1.390s
2022-05-18T04:24:01.8039569Z 
2022-05-18T04:24:01.8039642Z OK (skipped=1)
2022-05-18T04:24:01.8039752Z 
2022-05-18T04:24:01.8039838Z Generating XML reports...
2022-05-18T04:24:01.8071237Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042400.xml
2022-05-18T04:24:02.8367717Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:24:02.8377489Z 
2022-05-18T04:24:02.8377593Z Running tests...
2022-05-18T04:24:02.8378070Z ----------------------------------------------------------------------
2022-05-18T04:24:02.8397890Z   test_ddp_model_diff_num_params_across_ranks (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'nccl', 'gloo'} (0.002s)
2022-05-18T04:24:02.8398421Z 
2022-05-18T04:24:02.8398773Z ----------------------------------------------------------------------
2022-05-18T04:24:02.8399157Z Ran 1 test in 0.002s
2022-05-18T04:24:02.8399340Z 
2022-05-18T04:24:02.8399458Z OK (skipped=1)
2022-05-18T04:24:02.8399635Z 
2022-05-18T04:24:02.8399778Z Generating XML reports...
2022-05-18T04:24:02.8430900Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042402.xml
2022-05-18T04:24:03.6878786Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:24:03.6888606Z 
2022-05-18T04:24:03.6888757Z Running tests...
2022-05-18T04:24:03.6889227Z ----------------------------------------------------------------------
2022-05-18T04:24:03.6909855Z   test_ddp_model_diff_shape_across_ranks (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'nccl'} (0.002s)
2022-05-18T04:24:03.6910308Z 
2022-05-18T04:24:03.6910665Z ----------------------------------------------------------------------
2022-05-18T04:24:03.6911056Z Ran 1 test in 0.002s
2022-05-18T04:24:03.6911251Z 
2022-05-18T04:24:03.6911374Z OK (skipped=1)
2022-05-18T04:24:03.6911553Z 
2022-05-18T04:24:03.6911697Z Generating XML reports...
2022-05-18T04:24:03.6944162Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042403.xml
2022-05-18T04:24:04.5464301Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:24:04.5474300Z 
2022-05-18T04:24:04.5474453Z Running tests...
2022-05-18T04:24:04.5474961Z ----------------------------------------------------------------------
2022-05-18T04:24:04.5490910Z   test_ddp_multiple_nested_unused_params_err_ignore_params (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'nccl', 'gloo'} (0.001s)
2022-05-18T04:24:04.5491274Z 
2022-05-18T04:24:04.5491593Z ----------------------------------------------------------------------
2022-05-18T04:24:04.5491852Z Ran 1 test in 0.002s
2022-05-18T04:24:04.5492017Z 
2022-05-18T04:24:04.5492096Z OK (skipped=1)
2022-05-18T04:24:04.5492205Z 
2022-05-18T04:24:04.5492291Z Generating XML reports...
2022-05-18T04:24:04.5523449Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042404.xml
2022-05-18T04:24:05.3872418Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:24:05.3882100Z 
2022-05-18T04:24:05.3882295Z Running tests...
2022-05-18T04:24:05.3882947Z ----------------------------------------------------------------------
2022-05-18T04:24:05.3898325Z   test_ddp_multiple_nested_unused_params_error (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'nccl'} (0.001s)
2022-05-18T04:24:05.3898696Z 
2022-05-18T04:24:05.3898900Z ----------------------------------------------------------------------
2022-05-18T04:24:05.3899135Z Ran 1 test in 0.002s
2022-05-18T04:24:05.3899248Z 
2022-05-18T04:24:05.3899322Z OK (skipped=1)
2022-05-18T04:24:05.3899432Z 
2022-05-18T04:24:05.3899517Z Generating XML reports...
2022-05-18T04:24:05.3933214Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042405.xml
2022-05-18T04:24:06.2297092Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:24:06.2307407Z 
2022-05-18T04:24:06.2307555Z Running tests...
2022-05-18T04:24:06.2308014Z ----------------------------------------------------------------------
2022-05-18T04:24:06.2331573Z   test_ddp_namedtuple (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'nccl'} (0.002s)
2022-05-18T04:24:06.2331858Z 
2022-05-18T04:24:06.2332115Z ----------------------------------------------------------------------
2022-05-18T04:24:06.2332397Z Ran 1 test in 0.002s
2022-05-18T04:24:06.2332511Z 
2022-05-18T04:24:06.2332584Z OK (skipped=1)
2022-05-18T04:24:06.2332692Z 
2022-05-18T04:24:06.2332766Z Generating XML reports...
2022-05-18T04:24:06.2363327Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042406.xml
2022-05-18T04:24:07.0644145Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:24:07.0653616Z 
2022-05-18T04:24:07.0653747Z Running tests...
2022-05-18T04:24:07.0654175Z ----------------------------------------------------------------------
2022-05-18T04:24:07.3491682Z   test_ddp_new_tensor_in_fwd (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17247
2022-05-18T04:24:07.3513345Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17248
2022-05-18T04:24:07.3535507Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17249
2022-05-18T04:24:08.1711235Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:24:08.1812322Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:24:08.1812852Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:24:08.1813476Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:24:08.1814026Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:24:08.1814770Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:24:08.1822243Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:24:08.1823564Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:24:08.1824519Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:24:08.3583510Z skip: Need at least 2 CUDA devices (1.293s)
2022-05-18T04:24:08.3583821Z 
2022-05-18T04:24:08.3584210Z ----------------------------------------------------------------------
2022-05-18T04:24:08.3584483Z Ran 1 test in 1.293s
2022-05-18T04:24:08.3584599Z 
2022-05-18T04:24:08.3584672Z OK (skipped=1)
2022-05-18T04:24:08.3584780Z 
2022-05-18T04:24:08.3584867Z Generating XML reports...
2022-05-18T04:24:08.3615176Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042407.xml
2022-05-18T04:24:09.2899048Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:24:09.2909289Z 
2022-05-18T04:24:09.2909385Z Running tests...
2022-05-18T04:24:09.2910108Z ----------------------------------------------------------------------
2022-05-18T04:24:09.5718447Z   test_ddp_new_tensor_in_fwd_static_graph (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17300
2022-05-18T04:24:09.5740557Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17301
2022-05-18T04:24:09.5763568Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17302
2022-05-18T04:24:10.4305182Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:24:10.4365643Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:24:10.4366073Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:24:10.4366705Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:24:10.4367224Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:24:10.4405473Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:24:10.4474796Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:24:10.4475333Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:24:10.5418765Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:24:10.6813378Z skip: Need at least 2 CUDA devices (1.390s)
2022-05-18T04:24:10.6813657Z 
2022-05-18T04:24:10.6814117Z ----------------------------------------------------------------------
2022-05-18T04:24:10.6814525Z Ran 1 test in 1.390s
2022-05-18T04:24:10.6814703Z 
2022-05-18T04:24:10.6814820Z OK (skipped=1)
2022-05-18T04:24:10.6815010Z 
2022-05-18T04:24:10.6815139Z Generating XML reports...
2022-05-18T04:24:10.6846938Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042409.xml
2022-05-18T04:24:11.6021179Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:24:11.6031607Z 
2022-05-18T04:24:11.6031711Z Running tests...
2022-05-18T04:24:11.6032110Z ----------------------------------------------------------------------
2022-05-18T04:24:11.6047969Z   test_ddp_profiling_autograd_profiler (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'nccl'} (0.001s)
2022-05-18T04:24:11.6048640Z 
2022-05-18T04:24:11.6048897Z ----------------------------------------------------------------------
2022-05-18T04:24:11.6049144Z Ran 1 test in 0.002s
2022-05-18T04:24:11.6049246Z 
2022-05-18T04:24:11.6049407Z OK (skipped=1)
2022-05-18T04:24:11.6049516Z 
2022-05-18T04:24:11.6049604Z Generating XML reports...
2022-05-18T04:24:11.6080098Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042411.xml
2022-05-18T04:24:12.4380685Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:24:12.4390620Z 
2022-05-18T04:24:12.4390721Z Running tests...
2022-05-18T04:24:12.4391230Z ----------------------------------------------------------------------
2022-05-18T04:24:12.4408937Z   test_ddp_profiling_torch_profiler (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'nccl', 'gloo'} (0.002s)
2022-05-18T04:24:12.4409307Z 
2022-05-18T04:24:12.4409563Z ----------------------------------------------------------------------
2022-05-18T04:24:12.4409798Z Ran 1 test in 0.002s
2022-05-18T04:24:12.4409914Z 
2022-05-18T04:24:12.4409988Z OK (skipped=1)
2022-05-18T04:24:12.4410139Z 
2022-05-18T04:24:12.4410246Z Generating XML reports...
2022-05-18T04:24:12.4440727Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042412.xml
2022-05-18T04:24:13.2730545Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:24:13.2740098Z 
2022-05-18T04:24:13.2740304Z Running tests...
2022-05-18T04:24:13.2740767Z ----------------------------------------------------------------------
2022-05-18T04:24:13.5621785Z   test_ddp_python_error_logged (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17373
2022-05-18T04:24:13.5643931Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17374
2022-05-18T04:24:13.5667368Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17375
2022-05-18T04:24:14.3780838Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:24:14.3882310Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:24:14.3882720Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:24:14.3883344Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:24:14.3883861Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:24:14.3884386Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:24:14.3892660Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:24:14.3893681Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:24:14.3894547Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:24:14.5717301Z skip: Need at least 2 CUDA devices (1.297s)
2022-05-18T04:24:14.5717599Z 
2022-05-18T04:24:14.5718192Z ----------------------------------------------------------------------
2022-05-18T04:24:14.5718541Z Ran 1 test in 1.298s
2022-05-18T04:24:14.5718659Z 
2022-05-18T04:24:14.5718733Z OK (skipped=1)
2022-05-18T04:24:14.5718827Z 
2022-05-18T04:24:14.5718915Z Generating XML reports...
2022-05-18T04:24:14.5749406Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042413.xml
2022-05-18T04:24:15.5127404Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:24:15.5136827Z 
2022-05-18T04:24:15.5137188Z Running tests...
2022-05-18T04:24:15.5137587Z ----------------------------------------------------------------------
2022-05-18T04:24:15.7957065Z   test_ddp_returns_tensor_with_no_grad (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17426
2022-05-18T04:24:15.7978224Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17427
2022-05-18T04:24:15.8001423Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17428
2022-05-18T04:24:16.5838762Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:24:16.5839416Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:24:16.5839934Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:24:16.5840577Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:24:16.5841148Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:24:16.5841679Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:24:16.5849016Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:24:16.5849735Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:24:16.5851416Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:24:16.8049321Z skip: Need at least 2 CUDA devices (1.291s)
2022-05-18T04:24:16.8049591Z 
2022-05-18T04:24:16.8050121Z ----------------------------------------------------------------------
2022-05-18T04:24:16.8050533Z Ran 1 test in 1.291s
2022-05-18T04:24:16.8050748Z 
2022-05-18T04:24:16.8050912Z OK (skipped=1)
2022-05-18T04:24:16.8051101Z 
2022-05-18T04:24:16.8051246Z Generating XML reports...
2022-05-18T04:24:16.8084207Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042415.xml
2022-05-18T04:24:17.7308874Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:24:17.7319287Z 
2022-05-18T04:24:17.7319489Z Running tests...
2022-05-18T04:24:17.7320093Z ----------------------------------------------------------------------
2022-05-18T04:24:17.7343646Z   test_ddp_shared_grad_acc_unused_params (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'nccl', 'gloo'} (0.002s)
2022-05-18T04:24:17.7343995Z 
2022-05-18T04:24:17.7344212Z ----------------------------------------------------------------------
2022-05-18T04:24:17.7344449Z Ran 1 test in 0.002s
2022-05-18T04:24:17.7344568Z 
2022-05-18T04:24:17.7344643Z OK (skipped=1)
2022-05-18T04:24:17.7344765Z 
2022-05-18T04:24:17.7344853Z Generating XML reports...
2022-05-18T04:24:17.7375410Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042417.xml
2022-05-18T04:24:18.5664332Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:24:18.5674141Z 
2022-05-18T04:24:18.5674261Z Running tests...
2022-05-18T04:24:18.5674739Z ----------------------------------------------------------------------
2022-05-18T04:24:18.8486900Z   test_ddp_static_graph_nested_types (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77625 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure IN_CI is not set and you are not using the flag --import-disabled-tests. (0.281s)
2022-05-18T04:24:18.8487401Z 
2022-05-18T04:24:18.8487607Z ----------------------------------------------------------------------
2022-05-18T04:24:18.8488054Z Ran 1 test in 0.281s
2022-05-18T04:24:18.8488170Z 
2022-05-18T04:24:18.8488230Z OK (skipped=1)
2022-05-18T04:24:18.8488338Z 
2022-05-18T04:24:18.8488486Z Generating XML reports...
2022-05-18T04:24:18.8514398Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042418.xml
2022-05-18T04:24:19.7460725Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:24:19.7471466Z 
2022-05-18T04:24:19.7471951Z Running tests...
2022-05-18T04:24:19.7472385Z ----------------------------------------------------------------------
2022-05-18T04:24:20.0294486Z   test_ddp_sync_bn_training_vs_eval (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17499
2022-05-18T04:24:20.0316899Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17500
2022-05-18T04:24:20.0339927Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17501
2022-05-18T04:24:20.8887007Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:24:20.8887535Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:24:20.8887955Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:24:20.8888609Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:24:20.8889182Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:24:20.8889988Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:24:20.8897104Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:24:20.8897782Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:24:20.8898455Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:24:21.0389314Z skip: Need at least 2 CUDA devices (1.291s)
2022-05-18T04:24:21.0389492Z 
2022-05-18T04:24:21.0389814Z ----------------------------------------------------------------------
2022-05-18T04:24:21.0390118Z Ran 1 test in 1.292s
2022-05-18T04:24:21.0390234Z 
2022-05-18T04:24:21.0390296Z OK (skipped=1)
2022-05-18T04:24:21.0390405Z 
2022-05-18T04:24:21.0390493Z Generating XML reports...
2022-05-18T04:24:21.0420334Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042419.xml
2022-05-18T04:24:21.9623119Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:24:21.9633636Z 
2022-05-18T04:24:21.9634073Z Running tests...
2022-05-18T04:24:21.9634479Z ----------------------------------------------------------------------
2022-05-18T04:24:22.2458721Z   test_ddp_sync_module_states (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17552
2022-05-18T04:24:22.2481590Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17553
2022-05-18T04:24:22.2504686Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17554
2022-05-18T04:24:23.0607938Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:24:23.0709585Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:24:23.0710200Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:24:23.0711158Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:24:23.0712036Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:24:23.0712639Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:24:23.0719946Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:24:23.0721010Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:24:23.0722189Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:24:23.2554571Z skip: Need at least 2 CUDA devices (1.292s)
2022-05-18T04:24:23.2554810Z 
2022-05-18T04:24:23.2555133Z ----------------------------------------------------------------------
2022-05-18T04:24:23.2555384Z Ran 1 test in 1.292s
2022-05-18T04:24:23.2555498Z 
2022-05-18T04:24:23.2555575Z OK (skipped=1)
2022-05-18T04:24:23.2555754Z 
2022-05-18T04:24:23.2555843Z Generating XML reports...
2022-05-18T04:24:23.2587340Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042421.xml
2022-05-18T04:24:24.1818902Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:24:24.1829836Z 
2022-05-18T04:24:24.1830304Z Running tests...
2022-05-18T04:24:24.1830745Z ----------------------------------------------------------------------
2022-05-18T04:24:24.4638451Z   test_ddp_uneven_input_exception (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17605
2022-05-18T04:24:24.4660276Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17606
2022-05-18T04:24:24.4683184Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17607
2022-05-18T04:24:25.2669284Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:24:25.2670027Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:24:25.2670708Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:24:25.2671733Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:24:25.2672264Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:24:25.2672771Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:24:25.2680928Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:24:25.2683794Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:24:25.2684493Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:24:25.4732220Z skip: Need at least 2 CUDA devices (1.290s)
2022-05-18T04:24:25.4732887Z 
2022-05-18T04:24:25.4733818Z ----------------------------------------------------------------------
2022-05-18T04:24:25.4734101Z Ran 1 test in 1.290s
2022-05-18T04:24:25.4734220Z 
2022-05-18T04:24:25.4734299Z OK (skipped=1)
2022-05-18T04:24:25.4734407Z 
2022-05-18T04:24:25.4735253Z Generating XML reports...
2022-05-18T04:24:25.4764868Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042424.xml
2022-05-18T04:24:26.4052012Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:24:26.4060411Z 
2022-05-18T04:24:26.4060550Z Running tests...
2022-05-18T04:24:26.4061135Z ----------------------------------------------------------------------
2022-05-18T04:24:26.6883025Z   test_ddp_uneven_input_join_disable (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17658
2022-05-18T04:24:26.6906066Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17659
2022-05-18T04:24:26.6929206Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17660
2022-05-18T04:24:27.5047349Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:24:27.5048390Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:24:27.5049056Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:24:27.5049928Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:24:27.5050455Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:24:27.5148428Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:24:27.5158949Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:24:27.5160132Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:24:27.5160492Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:24:27.6976380Z skip: Need at least 2 CUDA devices (1.291s)
2022-05-18T04:24:27.6976636Z 
2022-05-18T04:24:27.6977131Z ----------------------------------------------------------------------
2022-05-18T04:24:27.6977578Z Ran 1 test in 1.292s
2022-05-18T04:24:27.6977777Z 
2022-05-18T04:24:27.6977864Z OK (skipped=1)
2022-05-18T04:24:27.6977959Z 
2022-05-18T04:24:27.6978045Z Generating XML reports...
2022-05-18T04:24:27.7008718Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042426.xml
2022-05-18T04:24:28.6221263Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:24:28.6231276Z 
2022-05-18T04:24:28.6231416Z Running tests...
2022-05-18T04:24:28.6232145Z ----------------------------------------------------------------------
2022-05-18T04:24:28.9001344Z   test_ddp_uneven_inputs (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/75648 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure IN_CI is not set and you are not using the flag --import-disabled-tests. (0.277s)
2022-05-18T04:24:28.9001841Z 
2022-05-18T04:24:28.9002048Z ----------------------------------------------------------------------
2022-05-18T04:24:28.9002295Z Ran 1 test in 0.277s
2022-05-18T04:24:28.9002410Z 
2022-05-18T04:24:28.9002485Z OK (skipped=1)
2022-05-18T04:24:28.9002610Z 
2022-05-18T04:24:28.9002695Z Generating XML reports...
2022-05-18T04:24:28.9029190Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042428.xml
2022-05-18T04:24:29.8000931Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:24:29.8010631Z 
2022-05-18T04:24:29.8010773Z Running tests...
2022-05-18T04:24:29.8011362Z ----------------------------------------------------------------------
2022-05-18T04:24:30.0836292Z   test_ddp_uneven_inputs_stop_iteration_sync_bn (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17721
2022-05-18T04:24:30.0858054Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17722
2022-05-18T04:24:30.0881494Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17723
2022-05-18T04:24:30.9359292Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:24:30.9459293Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:24:30.9460067Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:24:30.9460701Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:24:30.9461227Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:24:30.9461733Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:24:30.9568378Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:24:30.9568932Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:24:31.0473219Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:24:31.1931764Z skip: Need at least 2 CUDA devices (1.392s)
2022-05-18T04:24:31.1932065Z 
2022-05-18T04:24:31.1932592Z ----------------------------------------------------------------------
2022-05-18T04:24:31.1932845Z Ran 1 test in 1.392s
2022-05-18T04:24:31.1932960Z 
2022-05-18T04:24:31.1933033Z OK (skipped=1)
2022-05-18T04:24:31.1933140Z 
2022-05-18T04:24:31.1933225Z Generating XML reports...
2022-05-18T04:24:31.1964229Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042429.xml
2022-05-18T04:24:32.1223400Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:24:32.1233509Z 
2022-05-18T04:24:32.1233642Z Running tests...
2022-05-18T04:24:32.1234085Z ----------------------------------------------------------------------
2022-05-18T04:24:32.1263024Z   test_ddp_unused_params_rebuild_buckets_exception (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'nccl'} (0.003s)
2022-05-18T04:24:32.1263326Z 
2022-05-18T04:24:32.1263586Z ----------------------------------------------------------------------
2022-05-18T04:24:32.1263887Z Ran 1 test in 0.003s
2022-05-18T04:24:32.1264002Z 
2022-05-18T04:24:32.1264065Z OK (skipped=1)
2022-05-18T04:24:32.1264172Z 
2022-05-18T04:24:32.1264257Z Generating XML reports...
2022-05-18T04:24:32.1294690Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042432.xml
2022-05-18T04:24:32.9637720Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:24:32.9648097Z 
2022-05-18T04:24:32.9648191Z Running tests...
2022-05-18T04:24:32.9648594Z ----------------------------------------------------------------------
2022-05-18T04:24:33.2469503Z   test_destroy_full_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17784
2022-05-18T04:24:33.2491597Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17785
2022-05-18T04:24:33.2514155Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17786
2022-05-18T04:24:34.0306894Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:24:34.0363212Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:24:34.0363653Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:24:34.0364286Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:24:34.0364820Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:24:34.0407643Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:24:34.0472586Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:24:34.0472961Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:24:34.1419834Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:24:34.1585088Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:24:34.1686674Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:24:34.1687232Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-05-18T04:24:34.1687885Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:24:34.1688430Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:24:34.1688958Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:24:34.4567104Z ok (1.492s)
2022-05-18T04:24:34.4567341Z 
2022-05-18T04:24:34.4567873Z ----------------------------------------------------------------------
2022-05-18T04:24:34.4568226Z Ran 1 test in 1.492s
2022-05-18T04:24:34.4568339Z 
2022-05-18T04:24:34.4568402Z OK
2022-05-18T04:24:34.4568493Z 
2022-05-18T04:24:34.4568585Z Generating XML reports...
2022-05-18T04:24:34.4604674Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042432.xml
2022-05-18T04:24:35.3826421Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:24:35.3835380Z 
2022-05-18T04:24:35.3835670Z Running tests...
2022-05-18T04:24:35.3836263Z ----------------------------------------------------------------------
2022-05-18T04:24:35.6668005Z   test_destroy_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17846
2022-05-18T04:24:35.6689984Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17847
2022-05-18T04:24:35.6713161Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17848
2022-05-18T04:24:36.4829694Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:24:36.4901152Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:24:36.4901574Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:24:36.4902216Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:24:36.4902761Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:24:36.4930083Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:24:36.5010252Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:24:36.5011164Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:24:36.5217170Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:24:36.5217559Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-05-18T04:24:36.5941929Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:24:36.5943669Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:24:36.5944587Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:24:36.6026383Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:24:36.6026958Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:24:36.8766635Z ok (1.493s)
2022-05-18T04:24:36.8766799Z 
2022-05-18T04:24:36.8767115Z ----------------------------------------------------------------------
2022-05-18T04:24:36.8767469Z Ran 1 test in 1.493s
2022-05-18T04:24:36.8767583Z 
2022-05-18T04:24:36.8767644Z OK
2022-05-18T04:24:36.8767734Z 
2022-05-18T04:24:36.8767814Z Generating XML reports...
2022-05-18T04:24:36.8798024Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042435.xml
2022-05-18T04:24:37.7978479Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:24:37.7988554Z 
2022-05-18T04:24:37.7988675Z Running tests...
2022-05-18T04:24:37.7989278Z ----------------------------------------------------------------------
2022-05-18T04:24:38.0824577Z   test_detect_ddp_is_actually_static (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17905
2022-05-18T04:24:38.0846509Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17906
2022-05-18T04:24:38.0870106Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17907
2022-05-18T04:24:38.9470503Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:24:38.9570588Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:24:38.9571170Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:24:38.9572157Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:24:38.9573030Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:24:38.9573883Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:24:38.9679698Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:24:38.9680613Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:24:39.0584690Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:24:39.1922019Z skip: Need at least 2 CUDA devices (1.393s)
2022-05-18T04:24:39.1922304Z 
2022-05-18T04:24:39.1922812Z ----------------------------------------------------------------------
2022-05-18T04:24:39.1923080Z Ran 1 test in 1.393s
2022-05-18T04:24:39.1923180Z 
2022-05-18T04:24:39.1923253Z OK (skipped=1)
2022-05-18T04:24:39.1923362Z 
2022-05-18T04:24:39.1923462Z Generating XML reports...
2022-05-18T04:24:39.1953981Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042437.xml
2022-05-18T04:24:40.1208183Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:24:40.1217952Z 
2022-05-18T04:24:40.1218146Z Running tests...
2022-05-18T04:24:40.1218582Z ----------------------------------------------------------------------
2022-05-18T04:24:40.1237366Z   test_different_graph_across_ranks (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'nccl', 'gloo'} (0.002s)
2022-05-18T04:24:40.1237930Z 
2022-05-18T04:24:40.1238307Z ----------------------------------------------------------------------
2022-05-18T04:24:40.1238922Z Ran 1 test in 0.002s
2022-05-18T04:24:40.1239036Z 
2022-05-18T04:24:40.1239115Z OK (skipped=1)
2022-05-18T04:24:40.1239222Z 
2022-05-18T04:24:40.1239308Z Generating XML reports...
2022-05-18T04:24:40.1269381Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042440.xml
2022-05-18T04:24:40.9553348Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:24:40.9563269Z 
2022-05-18T04:24:40.9563374Z Running tests...
2022-05-18T04:24:40.9563792Z ----------------------------------------------------------------------
2022-05-18T04:24:41.2381655Z   test_dump_DDP_relevant_env_vars (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17968
2022-05-18T04:24:41.2404427Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17969
2022-05-18T04:24:41.2427306Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17970
2022-05-18T04:24:42.0244759Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:24:42.0345131Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:24:42.0345750Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:24:42.0346492Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:24:42.0347039Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:24:42.0347618Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:24:42.0452479Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:24:42.1358358Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:24:42.1358759Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:24:42.3479868Z ok (1.391s)
2022-05-18T04:24:42.3480128Z 
2022-05-18T04:24:42.3480630Z ----------------------------------------------------------------------
2022-05-18T04:24:42.3480949Z Ran 1 test in 1.392s
2022-05-18T04:24:42.3481065Z 
2022-05-18T04:24:42.3481125Z OK
2022-05-18T04:24:42.3481215Z 
2022-05-18T04:24:42.3481298Z Generating XML reports...
2022-05-18T04:24:42.3519708Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042440.xml
2022-05-18T04:24:43.2746974Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:24:43.2757233Z 
2022-05-18T04:24:43.2757612Z Running tests...
2022-05-18T04:24:43.2758242Z ----------------------------------------------------------------------
2022-05-18T04:24:43.5579835Z   test_gather (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18021
2022-05-18T04:24:43.5602553Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18022
2022-05-18T04:24:43.5625011Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18023
2022-05-18T04:24:44.3530779Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:24:44.3531470Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:24:44.3532070Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:24:44.3532751Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:24:44.3533288Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:24:44.3534016Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:24:44.3541439Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:24:44.3542172Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:24:44.3543815Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:24:44.6675818Z ok (1.392s)
2022-05-18T04:24:44.6676115Z 
2022-05-18T04:24:44.6676600Z ----------------------------------------------------------------------
2022-05-18T04:24:44.6676988Z Ran 1 test in 1.392s
2022-05-18T04:24:44.6677176Z 
2022-05-18T04:24:44.6677269Z OK
2022-05-18T04:24:44.6677423Z 
2022-05-18T04:24:44.6677614Z Generating XML reports...
2022-05-18T04:24:44.6709038Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042443.xml
2022-05-18T04:24:45.5945634Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:24:45.5955886Z 
2022-05-18T04:24:45.5956058Z Running tests...
2022-05-18T04:24:45.5956554Z ----------------------------------------------------------------------
2022-05-18T04:24:45.8797160Z   test_gather_checks (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18077
2022-05-18T04:24:45.8820055Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18078
2022-05-18T04:24:45.8843049Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18079
2022-05-18T04:24:46.6954621Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:24:46.7055153Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:24:46.7055776Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:24:46.7056659Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:24:46.7057237Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:24:46.7057752Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:24:46.7163810Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:24:46.8070426Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:24:46.8070812Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:24:46.9893255Z ok (1.393s)
2022-05-18T04:24:46.9893519Z 
2022-05-18T04:24:46.9894086Z ----------------------------------------------------------------------
2022-05-18T04:24:46.9894507Z Ran 1 test in 1.394s
2022-05-18T04:24:46.9894626Z 
2022-05-18T04:24:46.9894688Z OK
2022-05-18T04:24:46.9894783Z 
2022-05-18T04:24:46.9894873Z Generating XML reports...
2022-05-18T04:24:46.9926073Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042445.xml
2022-05-18T04:24:47.9156021Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:24:47.9165807Z 
2022-05-18T04:24:47.9165946Z Running tests...
2022-05-18T04:24:47.9166401Z ----------------------------------------------------------------------
2022-05-18T04:24:47.9181900Z   test_gather_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA gather (0.001s)
2022-05-18T04:24:47.9182200Z 
2022-05-18T04:24:47.9182569Z ----------------------------------------------------------------------
2022-05-18T04:24:47.9183119Z Ran 1 test in 0.002s
2022-05-18T04:24:47.9183508Z 
2022-05-18T04:24:47.9183584Z OK (skipped=1)
2022-05-18T04:24:47.9183694Z 
2022-05-18T04:24:47.9183784Z Generating XML reports...
2022-05-18T04:24:47.9214248Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042447.xml
2022-05-18T04:24:48.7448513Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:24:48.7457849Z 
2022-05-18T04:24:48.7458094Z Running tests...
2022-05-18T04:24:48.7458712Z ----------------------------------------------------------------------
2022-05-18T04:24:49.0254412Z   test_gather_full_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18140
2022-05-18T04:24:49.0276250Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18141
2022-05-18T04:24:49.0299037Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18142
2022-05-18T04:24:49.8116024Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:24:49.8116632Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:24:49.8117206Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:24:49.8118262Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:24:49.8119140Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:24:49.8119693Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:24:49.8125774Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:24:49.8126310Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:24:49.8127036Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:24:49.8234026Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:24:49.8234666Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:24:49.8235325Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-05-18T04:24:49.8236179Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:24:49.8236701Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:24:49.8239121Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:24:50.1349452Z ok (1.389s)
2022-05-18T04:24:50.1349850Z 
2022-05-18T04:24:50.1350364Z ----------------------------------------------------------------------
2022-05-18T04:24:50.1350671Z Ran 1 test in 1.389s
2022-05-18T04:24:50.1350813Z 
2022-05-18T04:24:50.1350917Z OK
2022-05-18T04:24:50.1351014Z 
2022-05-18T04:24:50.1351111Z Generating XML reports...
2022-05-18T04:24:50.1381217Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042448.xml
2022-05-18T04:24:51.0623205Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:24:51.0633059Z 
2022-05-18T04:24:51.0633264Z Running tests...
2022-05-18T04:24:51.0633671Z ----------------------------------------------------------------------
2022-05-18T04:24:51.3456064Z   test_gather_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18205
2022-05-18T04:24:51.3479274Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18206
2022-05-18T04:24:51.3502091Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18207
2022-05-18T04:24:52.1620795Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:24:52.1721923Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:24:52.1722380Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:24:52.1723026Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:24:52.1723563Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:24:52.1724072Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:24:52.1732843Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:24:52.1734509Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:24:52.1735364Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:24:52.1735961Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:24:52.1940580Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-05-18T04:24:52.1941181Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:24:52.1941918Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:24:52.1942450Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:24:52.2040800Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:24:52.5553447Z ok (1.492s)
2022-05-18T04:24:52.5553677Z 
2022-05-18T04:24:52.5554193Z ----------------------------------------------------------------------
2022-05-18T04:24:52.5554610Z Ran 1 test in 1.492s
2022-05-18T04:24:52.5554734Z 
2022-05-18T04:24:52.5554798Z OK
2022-05-18T04:24:52.5554892Z 
2022-05-18T04:24:52.5554974Z Generating XML reports...
2022-05-18T04:24:52.5587668Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042451.xml
2022-05-18T04:24:53.4793594Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:24:53.4803503Z 
2022-05-18T04:24:53.4803606Z Running tests...
2022-05-18T04:24:53.4804095Z ----------------------------------------------------------------------
2022-05-18T04:24:53.7625142Z   test_gather_object (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18266
2022-05-18T04:24:53.7647942Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18267
2022-05-18T04:24:53.7670813Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18268
2022-05-18T04:24:54.5796577Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:24:54.5897278Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:24:54.5897991Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:24:54.5898711Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:24:54.5899234Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:24:54.5899937Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:24:54.6007197Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:24:54.6007944Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:24:54.6909948Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:24:54.8720688Z ok (1.391s)
2022-05-18T04:24:54.8720944Z 
2022-05-18T04:24:54.8721475Z ----------------------------------------------------------------------
2022-05-18T04:24:54.8721905Z Ran 1 test in 1.392s
2022-05-18T04:24:54.8722026Z 
2022-05-18T04:24:54.8722075Z OK
2022-05-18T04:24:54.8722168Z 
2022-05-18T04:24:54.8722262Z Generating XML reports...
2022-05-18T04:24:54.8753206Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042453.xml
2022-05-18T04:24:55.7979885Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:24:55.7989903Z 
2022-05-18T04:24:55.7990025Z Running tests...
2022-05-18T04:24:55.7990412Z ----------------------------------------------------------------------
2022-05-18T04:24:56.0794453Z   test_gather_object_subgroup (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18319
2022-05-18T04:24:56.0817567Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18320
2022-05-18T04:24:56.0840187Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18321
2022-05-18T04:24:56.9012676Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:24:56.9113222Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:24:56.9113715Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:24:56.9114336Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:24:56.9114864Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:24:56.9115372Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:24:56.9222599Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:24:56.9223706Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:24:57.0125110Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:24:57.0637012Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:24:57.0739064Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:24:57.0739663Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-05-18T04:24:57.0740559Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:24:57.0741396Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:24:57.0742159Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:24:57.0882455Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-05-18T04:24:57.0983247Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 2
2022-05-18T04:24:57.0984009Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-05-18T04:24:57.0984731Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:3 with 3 nodes.
2022-05-18T04:24:57.0985268Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 3 nodes.
2022-05-18T04:24:57.0985792Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 3 nodes.
2022-05-18T04:24:57.1407304Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 0
2022-05-18T04:24:57.1408144Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 1
2022-05-18T04:24:57.1408845Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 2
2022-05-18T04:24:57.1409908Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:4 with 3 nodes.
2022-05-18T04:24:57.1410662Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:4 with 3 nodes.
2022-05-18T04:24:57.1411190Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:4 with 3 nodes.
2022-05-18T04:24:57.3894740Z ok (1.590s)
2022-05-18T04:24:57.3894900Z 
2022-05-18T04:24:57.3895304Z ----------------------------------------------------------------------
2022-05-18T04:24:57.3895597Z Ran 1 test in 1.590s
2022-05-18T04:24:57.3895715Z 
2022-05-18T04:24:57.3895779Z OK
2022-05-18T04:24:57.3895874Z 
2022-05-18T04:24:57.3895977Z Generating XML reports...
2022-05-18T04:24:57.3928014Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042455.xml
2022-05-18T04:24:58.3119365Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:24:58.3129132Z 
2022-05-18T04:24:58.3129225Z Running tests...
2022-05-18T04:24:58.3130119Z ----------------------------------------------------------------------
2022-05-18T04:24:58.5935849Z   test_get_backend (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18408
2022-05-18T04:24:58.5958206Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18409
2022-05-18T04:24:58.5981761Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18410
2022-05-18T04:24:59.3979273Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:24:59.4080003Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:24:59.4080733Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:24:59.4081684Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:24:59.4082500Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:24:59.4083033Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:24:59.4187425Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:24:59.5093633Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:24:59.5094221Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:24:59.5094971Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:24:59.5298900Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:24:59.5299879Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-05-18T04:24:59.5300580Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:24:59.5301114Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:24:59.5301628Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:24:59.7032637Z ok (1.390s)
2022-05-18T04:24:59.7033043Z 
2022-05-18T04:24:59.7033619Z ----------------------------------------------------------------------
2022-05-18T04:24:59.7033890Z Ran 1 test in 1.390s
2022-05-18T04:24:59.7034009Z 
2022-05-18T04:24:59.7034073Z OK
2022-05-18T04:24:59.7034165Z 
2022-05-18T04:24:59.7034260Z Generating XML reports...
2022-05-18T04:24:59.7064066Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042458.xml
2022-05-18T04:25:00.6265550Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:25:00.6275990Z 
2022-05-18T04:25:00.6276117Z Running tests...
2022-05-18T04:25:00.6276671Z ----------------------------------------------------------------------
2022-05-18T04:25:00.9088959Z   test_get_future (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18467
2022-05-18T04:25:00.9111284Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18468
2022-05-18T04:25:00.9132919Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18469
2022-05-18T04:25:01.7155059Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:25:01.7256475Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:25:01.7257334Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:25:01.7257785Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:25:01.7258279Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:25:01.7258815Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:25:01.7266243Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:25:01.7267195Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:25:01.7268577Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:25:01.9182989Z ok (1.290s)
2022-05-18T04:25:01.9183257Z 
2022-05-18T04:25:01.9183598Z ----------------------------------------------------------------------
2022-05-18T04:25:01.9183852Z Ran 1 test in 1.291s
2022-05-18T04:25:01.9183969Z 
2022-05-18T04:25:01.9184034Z OK
2022-05-18T04:25:01.9184119Z 
2022-05-18T04:25:01.9184217Z Generating XML reports...
2022-05-18T04:25:01.9216357Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042500.xml
2022-05-18T04:25:02.8434015Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:25:02.8444403Z 
2022-05-18T04:25:02.8444508Z Running tests...
2022-05-18T04:25:02.8445077Z ----------------------------------------------------------------------
2022-05-18T04:25:03.1255111Z   test_get_rank (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18520
2022-05-18T04:25:03.1277244Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18521
2022-05-18T04:25:03.1300202Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18522
2022-05-18T04:25:03.9413075Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:25:03.9454789Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:25:03.9455239Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:25:03.9455863Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:25:03.9456398Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:25:03.9513753Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:25:03.9564295Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:25:03.9564686Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:25:04.0524448Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:25:04.4353381Z ok (1.591s)
2022-05-18T04:25:04.4353672Z 
2022-05-18T04:25:04.4354204Z ----------------------------------------------------------------------
2022-05-18T04:25:04.4354591Z Ran 1 test in 1.591s
2022-05-18T04:25:04.4354708Z 
2022-05-18T04:25:04.4354770Z OK
2022-05-18T04:25:04.4354865Z 
2022-05-18T04:25:04.4354948Z Generating XML reports...
2022-05-18T04:25:04.4385842Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042502.xml
2022-05-18T04:25:05.3595044Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:25:05.3604691Z 
2022-05-18T04:25:05.3604788Z Running tests...
2022-05-18T04:25:05.3605341Z ----------------------------------------------------------------------
2022-05-18T04:25:05.6419808Z   test_get_rank_size_full_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18573
2022-05-18T04:25:05.6441765Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18574
2022-05-18T04:25:05.6464810Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18575
2022-05-18T04:25:06.4284000Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:25:06.4353878Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:25:06.4354425Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:25:06.4355032Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:25:06.4355580Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:25:06.4384578Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:25:06.4463538Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:25:06.4463947Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:25:06.5395822Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:25:06.5702403Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:25:06.5804132Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-05-18T04:25:06.5804684Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:25:06.5805518Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:25:06.5806105Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:25:06.5806625Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:25:06.8518565Z ok (1.491s)
2022-05-18T04:25:06.8518815Z 
2022-05-18T04:25:06.8519325Z ----------------------------------------------------------------------
2022-05-18T04:25:06.8519756Z Ran 1 test in 1.491s
2022-05-18T04:25:06.8519879Z 
2022-05-18T04:25:06.8519927Z OK
2022-05-18T04:25:06.8520019Z 
2022-05-18T04:25:06.8520111Z Generating XML reports...
2022-05-18T04:25:06.8550109Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042505.xml
2022-05-18T04:25:07.7654447Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:25:07.7664362Z 
2022-05-18T04:25:07.7664495Z Running tests...
2022-05-18T04:25:07.7665088Z ----------------------------------------------------------------------
2022-05-18T04:25:08.0514886Z   test_get_rank_size_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18635
2022-05-18T04:25:08.0537754Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18636
2022-05-18T04:25:08.0560469Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18637
2022-05-18T04:25:08.8743400Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:25:08.8844088Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:25:08.8844560Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:25:08.8845195Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:25:08.8845782Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:25:08.8846601Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:25:08.8854157Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:25:08.8855882Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:25:08.8856550Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:25:08.8857232Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:25:08.9062026Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:25:08.9064049Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-05-18T04:25:08.9064894Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:25:08.9065445Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:25:08.9160705Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:25:09.1611998Z ok (1.394s)
2022-05-18T04:25:09.1612243Z 
2022-05-18T04:25:09.1612749Z ----------------------------------------------------------------------
2022-05-18T04:25:09.1613023Z Ran 1 test in 1.395s
2022-05-18T04:25:09.1613139Z 
2022-05-18T04:25:09.1613200Z OK
2022-05-18T04:25:09.1613291Z 
2022-05-18T04:25:09.1613385Z Generating XML reports...
2022-05-18T04:25:09.1643401Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042507.xml
2022-05-18T04:25:10.0850201Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:25:10.0859886Z 
2022-05-18T04:25:10.0860012Z Running tests...
2022-05-18T04:25:10.0860536Z ----------------------------------------------------------------------
2022-05-18T04:25:10.0884672Z   test_invalid_static_graph (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'nccl'} (0.002s)
2022-05-18T04:25:10.0885040Z 
2022-05-18T04:25:10.0885299Z ----------------------------------------------------------------------
2022-05-18T04:25:10.0885585Z Ran 1 test in 0.002s
2022-05-18T04:25:10.0885700Z 
2022-05-18T04:25:10.0885775Z OK (skipped=1)
2022-05-18T04:25:10.0885882Z 
2022-05-18T04:25:10.0885969Z Generating XML reports...
2022-05-18T04:25:10.0917402Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042510.xml
2022-05-18T04:25:10.9235250Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:25:10.9245312Z 
2022-05-18T04:25:10.9245444Z Running tests...
2022-05-18T04:25:10.9245853Z ----------------------------------------------------------------------
2022-05-18T04:25:11.2083836Z   test_irecv (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18704
2022-05-18T04:25:11.2108944Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18705
2022-05-18T04:25:11.2133484Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18706
2022-05-18T04:25:12.0053909Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:25:12.0154362Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:25:12.0155105Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:25:12.0156064Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:25:12.0156598Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:25:12.0157109Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:25:12.0262568Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:25:12.1167644Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:25:12.1168021Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:25:12.4185077Z ok (1.494s)
2022-05-18T04:25:12.4185316Z 
2022-05-18T04:25:12.4185680Z ----------------------------------------------------------------------
2022-05-18T04:25:12.4185982Z Ran 1 test in 1.494s
2022-05-18T04:25:12.4186116Z 
2022-05-18T04:25:12.4186178Z OK
2022-05-18T04:25:12.4186305Z 
2022-05-18T04:25:12.4186420Z Generating XML reports...
2022-05-18T04:25:12.4224494Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042510.xml
2022-05-18T04:25:13.3404261Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:25:13.3413957Z 
2022-05-18T04:25:13.3414092Z Running tests...
2022-05-18T04:25:13.3414703Z ----------------------------------------------------------------------
2022-05-18T04:25:13.6219090Z   test_isend (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18757
2022-05-18T04:25:13.6241991Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18758
2022-05-18T04:25:13.6264891Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18759
2022-05-18T04:25:14.4089731Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:25:14.4190503Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:25:14.4190981Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:25:14.4191607Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:25:14.4192158Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:25:14.4192665Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:25:14.4200644Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:25:14.4201245Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:25:14.4203442Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:25:14.7316858Z ok (1.390s)
2022-05-18T04:25:14.7317448Z 
2022-05-18T04:25:14.7318094Z ----------------------------------------------------------------------
2022-05-18T04:25:14.7318570Z Ran 1 test in 1.390s
2022-05-18T04:25:14.7318767Z 
2022-05-18T04:25:14.7318848Z OK
2022-05-18T04:25:14.7318952Z 
2022-05-18T04:25:14.7319044Z Generating XML reports...
2022-05-18T04:25:14.7348546Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042513.xml
2022-05-18T04:25:15.6579743Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:25:15.6591100Z 
2022-05-18T04:25:15.6591249Z Running tests...
2022-05-18T04:25:15.6591671Z ----------------------------------------------------------------------
2022-05-18T04:25:15.9433878Z   test_isend_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18810
2022-05-18T04:25:15.9456903Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18811
2022-05-18T04:25:15.9480656Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18812
2022-05-18T04:25:16.7282460Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:25:16.7382450Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:25:16.7383245Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:25:16.7384138Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:25:16.7384987Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:25:16.7385917Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:25:16.7491521Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:25:16.7491996Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:25:16.8397308Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:25:17.1533562Z ok (1.494s)
2022-05-18T04:25:17.1533790Z 
2022-05-18T04:25:17.1534253Z ----------------------------------------------------------------------
2022-05-18T04:25:17.1534650Z Ran 1 test in 1.494s
2022-05-18T04:25:17.1534827Z 
2022-05-18T04:25:17.1534908Z OK
2022-05-18T04:25:17.1535061Z 
2022-05-18T04:25:17.1535207Z Generating XML reports...
2022-05-18T04:25:17.1566740Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042515.xml
2022-05-18T04:25:18.0787722Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:25:18.0797380Z 
2022-05-18T04:25:18.0797479Z Running tests...
2022-05-18T04:25:18.0797903Z ----------------------------------------------------------------------
2022-05-18T04:25:18.3593634Z   test_isend_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18866
2022-05-18T04:25:18.3615869Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18867
2022-05-18T04:25:18.3638527Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18868
2022-05-18T04:25:19.1774987Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:25:19.1876637Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:25:19.1877542Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:25:19.1877962Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:25:19.1878728Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:25:19.1879550Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:25:19.1887584Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:25:19.1888138Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:25:19.1889555Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:25:19.5690418Z ok (1.489s)
2022-05-18T04:25:19.5690631Z 
2022-05-18T04:25:19.5691059Z ----------------------------------------------------------------------
2022-05-18T04:25:19.5691374Z Ran 1 test in 1.489s
2022-05-18T04:25:19.5691497Z 
2022-05-18T04:25:19.5691561Z OK
2022-05-18T04:25:19.5691655Z 
2022-05-18T04:25:19.5691735Z Generating XML reports...
2022-05-18T04:25:19.5724121Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042518.xml
2022-05-18T04:25:20.4947544Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:25:20.4957408Z 
2022-05-18T04:25:20.4957535Z Running tests...
2022-05-18T04:25:20.4957945Z ----------------------------------------------------------------------
2022-05-18T04:25:20.4973386Z   test_monitored_barrier_allreduce_hang (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'nccl', 'gloo'} (0.001s)
2022-05-18T04:25:20.4973753Z 
2022-05-18T04:25:20.4974074Z ----------------------------------------------------------------------
2022-05-18T04:25:20.4974332Z Ran 1 test in 0.002s
2022-05-18T04:25:20.4974447Z 
2022-05-18T04:25:20.4974533Z OK (skipped=1)
2022-05-18T04:25:20.4974643Z 
2022-05-18T04:25:20.4974743Z Generating XML reports...
2022-05-18T04:25:20.5005469Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042520.xml
2022-05-18T04:25:21.3296104Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:25:21.3306160Z 
2022-05-18T04:25:21.3306370Z Running tests...
2022-05-18T04:25:21.3306746Z ----------------------------------------------------------------------
2022-05-18T04:25:21.3322325Z   test_monitored_barrier_allreduce_hang_wait_all_ranks (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'nccl'} (0.001s)
2022-05-18T04:25:21.3322676Z 
2022-05-18T04:25:21.3323187Z ----------------------------------------------------------------------
2022-05-18T04:25:21.3323422Z Ran 1 test in 0.002s
2022-05-18T04:25:21.3323537Z 
2022-05-18T04:25:21.3323613Z OK (skipped=1)
2022-05-18T04:25:21.3323805Z 
2022-05-18T04:25:21.3323920Z Generating XML reports...
2022-05-18T04:25:21.3353840Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042521.xml
2022-05-18T04:25:22.1630325Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:25:22.1640783Z 
2022-05-18T04:25:22.1640887Z Running tests...
2022-05-18T04:25:22.1641303Z ----------------------------------------------------------------------
2022-05-18T04:25:22.4501994Z   test_monitored_barrier_failure_order (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18942
2022-05-18T04:25:22.4524513Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18943
2022-05-18T04:25:22.4547125Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18944
2022-05-18T04:25:23.2555488Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:25:23.2655953Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:25:23.2656557Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:25:23.2657176Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:25:23.2657705Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:25:23.2658224Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:25:23.2765294Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:25:23.2765832Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:25:23.3668102Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:25:25.3689748Z [E ProcessGroupGloo.cpp:136] Rank 1 successfully reached monitoredBarrier, but received errors while waiting for send/recv from rank 0. Please check rank 0 logs for faulty rank.
2022-05-18T04:25:25.3787784Z [E ProcessGroupGloo.cpp:136] [Rank 0]: Rank 2 failed to pass monitoredBarrier in 2000 ms
2022-05-18T04:25:25.6628551Z ok (3.498s)
2022-05-18T04:25:25.6628831Z 
2022-05-18T04:25:25.6629297Z ----------------------------------------------------------------------
2022-05-18T04:25:25.6629674Z Ran 1 test in 3.499s
2022-05-18T04:25:25.6629859Z 
2022-05-18T04:25:25.6629959Z OK
2022-05-18T04:25:25.6630098Z 
2022-05-18T04:25:25.6630242Z Generating XML reports...
2022-05-18T04:25:25.6661178Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042522.xml
2022-05-18T04:25:26.6070832Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:25:26.6080523Z 
2022-05-18T04:25:26.6080619Z Running tests...
2022-05-18T04:25:26.6081032Z ----------------------------------------------------------------------
2022-05-18T04:25:26.8950737Z   test_monitored_barrier_gloo (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18995
2022-05-18T04:25:26.8972840Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18996
2022-05-18T04:25:26.8996629Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18997
2022-05-18T04:25:27.7186833Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:25:27.7286042Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:25:27.7286627Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:25:27.7287422Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:25:27.7287967Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:25:27.7288496Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:25:27.7395989Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:25:27.7396581Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:25:27.8300905Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:25:29.8498927Z [E ProcessGroupGloo.cpp:136] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 2000 ms
2022-05-18T04:25:29.8499442Z [E ProcessGroupGloo.cpp:136] Rank 2 successfully reached monitoredBarrier, but received errors while waiting for send/recv from rank 0. Please check rank 0 logs for faulty rank.
2022-05-18T04:25:30.1076940Z ok (3.499s)
2022-05-18T04:25:30.1077238Z 
2022-05-18T04:25:30.1077723Z ----------------------------------------------------------------------
2022-05-18T04:25:30.1078165Z Ran 1 test in 3.500s
2022-05-18T04:25:30.1078346Z 
2022-05-18T04:25:30.1078412Z OK
2022-05-18T04:25:30.1078505Z 
2022-05-18T04:25:30.1078600Z Generating XML reports...
2022-05-18T04:25:30.1109079Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042526.xml
2022-05-18T04:25:31.0462140Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:25:31.0471882Z 
2022-05-18T04:25:31.0472019Z Running tests...
2022-05-18T04:25:31.0472452Z ----------------------------------------------------------------------
2022-05-18T04:25:31.3287095Z   test_monitored_barrier_gloo_rank_0_timeout (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19048
2022-05-18T04:25:31.3308119Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19049
2022-05-18T04:25:31.3330722Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19050
2022-05-18T04:25:32.1329585Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:25:32.1335467Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:25:32.1335991Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:25:32.1336616Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:25:32.1337150Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:25:32.1430164Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:25:32.1444993Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:25:32.1445611Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:25:32.2440551Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:25:32.2749510Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:25:32.2750022Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:25:32.2750449Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-05-18T04:25:32.2751313Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:25:32.2751923Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:25:32.2752579Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:25:32.2753202Z [E ProcessGroupGloo.cpp:136] Rank 0 timed out in monitoredBarrier after 0 ms.
2022-05-18T04:25:32.2753539Z No ranks successfully processed in monitoredBarrier.
2022-05-18T04:25:32.2778378Z [E ProcessGroupGloo.cpp:136] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 0 ms
2022-05-18T04:25:32.5381092Z ok (1.491s)
2022-05-18T04:25:32.5381322Z 
2022-05-18T04:25:32.5381771Z ----------------------------------------------------------------------
2022-05-18T04:25:32.5382200Z Ran 1 test in 1.491s
2022-05-18T04:25:32.5382386Z 
2022-05-18T04:25:32.5382481Z OK
2022-05-18T04:25:32.5382617Z 
2022-05-18T04:25:32.5382756Z Generating XML reports...
2022-05-18T04:25:32.5414637Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042531.xml
2022-05-18T04:25:33.4609098Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:25:33.4619533Z 
2022-05-18T04:25:33.4620008Z Running tests...
2022-05-18T04:25:33.4620414Z ----------------------------------------------------------------------
2022-05-18T04:25:33.7420453Z   test_monitored_barrier_gloo_subgroup (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19110
2022-05-18T04:25:33.7442448Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19111
2022-05-18T04:25:33.7465796Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19112
2022-05-18T04:25:34.5498113Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:25:34.5498775Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:25:34.5499133Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:25:34.5499752Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:25:34.5500280Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:25:34.5500802Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:25:34.5605399Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:25:34.6509297Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:25:34.6509990Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:25:34.6512785Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-05-18T04:25:34.6616290Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:25:34.6617060Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:25:34.6617703Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:25:34.6618238Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:25:34.6618745Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:25:34.6621642Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:279: UserWarning: Running monitored_barrier on global rank 2 which does not belong to the given group.
2022-05-18T04:25:34.6622150Z   f"Running {op_name} on global rank {global_rank} which does not "
2022-05-18T04:25:34.7721263Z [E ProcessGroupGloo.cpp:136] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 100 ms
2022-05-18T04:25:35.0519140Z ok (1.590s)
2022-05-18T04:25:35.0519362Z 
2022-05-18T04:25:35.0520197Z ----------------------------------------------------------------------
2022-05-18T04:25:35.0520478Z Ran 1 test in 1.590s
2022-05-18T04:25:35.0520598Z 
2022-05-18T04:25:35.0520692Z OK
2022-05-18T04:25:35.0520821Z 
2022-05-18T04:25:35.0520902Z Generating XML reports...
2022-05-18T04:25:35.0551838Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042533.xml
2022-05-18T04:25:35.9737076Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:25:35.9747637Z 
2022-05-18T04:25:35.9748208Z Running tests...
2022-05-18T04:25:35.9748617Z ----------------------------------------------------------------------
2022-05-18T04:25:36.2548454Z   test_monitored_barrier_wait_all_ranks (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19169
2022-05-18T04:25:36.2570808Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19170
2022-05-18T04:25:36.2593648Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19171
2022-05-18T04:25:37.0620390Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:25:37.0720344Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:25:37.0721026Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:25:37.0722070Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:25:37.0722891Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:25:37.0723744Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:25:37.0830044Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:25:37.0831356Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:25:37.1734262Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:25:37.2932888Z [E ProcessGroupGloo.cpp:2791] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 100 ms
2022-05-18T04:25:37.2933299Z [E ProcessGroupGloo.cpp:2791] [Rank 0]: Rank 2 failed to pass monitoredBarrier in 100 ms
2022-05-18T04:25:37.2933719Z [E ProcessGroupGloo.cpp:136] [Rank 0]: Ranks 1, 2 failed to pass monitoredBarrier in 100 ms
2022-05-18T04:25:37.5647179Z ok (1.590s)
2022-05-18T04:25:37.5647367Z 
2022-05-18T04:25:37.5647888Z ----------------------------------------------------------------------
2022-05-18T04:25:37.5648314Z Ran 1 test in 1.590s
2022-05-18T04:25:37.5648446Z 
2022-05-18T04:25:37.5648507Z OK
2022-05-18T04:25:37.5648596Z 
2022-05-18T04:25:37.5648687Z Generating XML reports...
2022-05-18T04:25:37.5678692Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042535.xml
2022-05-18T04:25:38.4845222Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:25:38.4855613Z 
2022-05-18T04:25:38.4855799Z Running tests...
2022-05-18T04:25:38.4856230Z ----------------------------------------------------------------------
2022-05-18T04:25:38.4877164Z   test_nccl_backend_bool_allgather (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl'} (0.002s)
2022-05-18T04:25:38.4877968Z 
2022-05-18T04:25:38.4878248Z ----------------------------------------------------------------------
2022-05-18T04:25:38.4878503Z Ran 1 test in 0.002s
2022-05-18T04:25:38.4878619Z 
2022-05-18T04:25:38.4878694Z OK (skipped=1)
2022-05-18T04:25:38.4878803Z 
2022-05-18T04:25:38.4878889Z Generating XML reports...
2022-05-18T04:25:38.4909042Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042538.xml
2022-05-18T04:25:39.3212451Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:25:39.3223092Z 
2022-05-18T04:25:39.3223203Z Running tests...
2022-05-18T04:25:39.3223791Z ----------------------------------------------------------------------
2022-05-18T04:25:39.3245748Z   test_nccl_backend_bool_allreduce (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl'} (0.002s)
2022-05-18T04:25:39.3246192Z 
2022-05-18T04:25:39.3246567Z ----------------------------------------------------------------------
2022-05-18T04:25:39.3246970Z Ran 1 test in 0.002s
2022-05-18T04:25:39.3247168Z 
2022-05-18T04:25:39.3247288Z OK (skipped=1)
2022-05-18T04:25:39.3247448Z 
2022-05-18T04:25:39.3247592Z Generating XML reports...
2022-05-18T04:25:39.3279336Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042539.xml
2022-05-18T04:25:40.1581531Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:25:40.1591507Z 
2022-05-18T04:25:40.1591611Z Running tests...
2022-05-18T04:25:40.1592157Z ----------------------------------------------------------------------
2022-05-18T04:25:40.1611806Z   test_nccl_backend_bool_broadcast (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl'} (0.002s)
2022-05-18T04:25:40.1612244Z 
2022-05-18T04:25:40.1612598Z ----------------------------------------------------------------------
2022-05-18T04:25:40.1613004Z Ran 1 test in 0.002s
2022-05-18T04:25:40.1613192Z 
2022-05-18T04:25:40.1613324Z OK (skipped=1)
2022-05-18T04:25:40.1613502Z 
2022-05-18T04:25:40.1613643Z Generating XML reports...
2022-05-18T04:25:40.1644699Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042540.xml
2022-05-18T04:25:40.9945693Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:25:40.9956257Z 
2022-05-18T04:25:40.9956568Z Running tests...
2022-05-18T04:25:40.9957268Z ----------------------------------------------------------------------
2022-05-18T04:25:40.9981431Z   test_nccl_backend_bool_reduce (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl'} (0.002s)
2022-05-18T04:25:40.9981852Z 
2022-05-18T04:25:40.9982206Z ----------------------------------------------------------------------
2022-05-18T04:25:40.9982634Z Ran 1 test in 0.003s
2022-05-18T04:25:40.9982820Z 
2022-05-18T04:25:40.9983113Z OK (skipped=1)
2022-05-18T04:25:40.9983274Z 
2022-05-18T04:25:40.9983429Z Generating XML reports...
2022-05-18T04:25:41.0022329Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042540.xml
2022-05-18T04:25:41.8340635Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:25:41.8351422Z 
2022-05-18T04:25:41.8351690Z Running tests...
2022-05-18T04:25:41.8372458Z ----------------------------------------------------------------------
2022-05-18T04:25:41.8373189Z   test_nccl_high_priority_stream (__main__.TestDistBackendWithSpawn) ... skip: Only NCCL backend supports high priority stream (0.002s)
2022-05-18T04:25:41.8373445Z 
2022-05-18T04:25:41.8373649Z ----------------------------------------------------------------------
2022-05-18T04:25:41.8373877Z Ran 1 test in 0.002s
2022-05-18T04:25:41.8374228Z 
2022-05-18T04:25:41.8374301Z OK (skipped=1)
2022-05-18T04:25:41.8374410Z 
2022-05-18T04:25:41.8374496Z Generating XML reports...
2022-05-18T04:25:41.8405257Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042541.xml
2022-05-18T04:25:42.6725558Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:25:42.6735657Z 
2022-05-18T04:25:42.6736063Z Running tests...
2022-05-18T04:25:42.6736485Z ----------------------------------------------------------------------
2022-05-18T04:25:42.6753826Z   test_new_subgroups (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2022-05-18T04:25:42.6754468Z 
2022-05-18T04:25:42.6754928Z ----------------------------------------------------------------------
2022-05-18T04:25:42.6755390Z Ran 1 test in 0.002s
2022-05-18T04:25:42.6755615Z 
2022-05-18T04:25:42.6755748Z OK (skipped=1)
2022-05-18T04:25:42.6755874Z 
2022-05-18T04:25:42.6755960Z Generating XML reports...
2022-05-18T04:25:42.6786487Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042542.xml
2022-05-18T04:25:43.5125865Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:25:43.5136869Z 
2022-05-18T04:25:43.5137316Z Running tests...
2022-05-18T04:25:43.5137726Z ----------------------------------------------------------------------
2022-05-18T04:25:43.5158636Z   test_new_subgroups_by_enumeration (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2022-05-18T04:25:43.5158986Z 
2022-05-18T04:25:43.5159291Z ----------------------------------------------------------------------
2022-05-18T04:25:43.5159542Z Ran 1 test in 0.002s
2022-05-18T04:25:43.5159656Z 
2022-05-18T04:25:43.5159730Z OK (skipped=1)
2022-05-18T04:25:43.5159824Z 
2022-05-18T04:25:43.5159910Z Generating XML reports...
2022-05-18T04:25:43.5190893Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042543.xml
2022-05-18T04:25:44.3549201Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:25:44.3559729Z 
2022-05-18T04:25:44.3559855Z Running tests...
2022-05-18T04:25:44.3560449Z ----------------------------------------------------------------------
2022-05-18T04:25:44.3579395Z   test_new_subgroups_by_enumeration_input_rank_exceeds_world_size (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2022-05-18T04:25:44.3579788Z 
2022-05-18T04:25:44.3580088Z ----------------------------------------------------------------------
2022-05-18T04:25:44.3589154Z Ran 1 test in 0.002s
2022-05-18T04:25:44.3589277Z 
2022-05-18T04:25:44.3589367Z OK (skipped=1)
2022-05-18T04:25:44.3589488Z 
2022-05-18T04:25:44.3589577Z Generating XML reports...
2022-05-18T04:25:44.3612156Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042544.xml
2022-05-18T04:25:45.1857696Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:25:45.1867745Z 
2022-05-18T04:25:45.1867853Z Running tests...
2022-05-18T04:25:45.1868691Z ----------------------------------------------------------------------
2022-05-18T04:25:45.4680656Z   test_new_subgroups_by_enumeration_negative_input_rank (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19302
2022-05-18T04:25:45.4702838Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19303
2022-05-18T04:25:45.4725608Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19304
2022-05-18T04:25:46.2657276Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:25:46.2658000Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:25:46.2658603Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:25:46.2659304Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:25:46.2659840Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:25:46.2660350Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:25:46.2764416Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:25:46.3671421Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:25:46.3671815Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:25:46.5776186Z skip: CUDA is not available. (1.391s)
2022-05-18T04:25:46.5776497Z 
2022-05-18T04:25:46.5777012Z ----------------------------------------------------------------------
2022-05-18T04:25:46.5777285Z Ran 1 test in 1.391s
2022-05-18T04:25:46.5777426Z 
2022-05-18T04:25:46.5777500Z OK (skipped=1)
2022-05-18T04:25:46.5777596Z 
2022-05-18T04:25:46.5777681Z Generating XML reports...
2022-05-18T04:25:46.5808036Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042545.xml
2022-05-18T04:25:47.4991711Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:25:47.5001903Z 
2022-05-18T04:25:47.5002014Z Running tests...
2022-05-18T04:25:47.5002587Z ----------------------------------------------------------------------
2022-05-18T04:25:47.7833672Z   test_new_subgroups_group_size_exceeds_world_size (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19355
2022-05-18T04:25:47.7855208Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19356
2022-05-18T04:25:47.7878323Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19357
2022-05-18T04:25:48.5906283Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:25:48.6006967Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:25:48.6007563Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:25:48.6008206Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:25:48.6008720Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:25:48.6009565Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:25:48.6114803Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:25:48.7022006Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:25:48.7022557Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:25:48.8929243Z skip: CUDA is not available. (1.392s)
2022-05-18T04:25:48.8929552Z 
2022-05-18T04:25:48.8929970Z ----------------------------------------------------------------------
2022-05-18T04:25:48.8930212Z Ran 1 test in 1.393s
2022-05-18T04:25:48.8930333Z 
2022-05-18T04:25:48.8930410Z OK (skipped=1)
2022-05-18T04:25:48.8930519Z 
2022-05-18T04:25:48.8930609Z Generating XML reports...
2022-05-18T04:25:48.8961038Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042547.xml
2022-05-18T04:25:49.8152858Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:25:49.8162273Z 
2022-05-18T04:25:49.8162419Z Running tests...
2022-05-18T04:25:49.8163209Z ----------------------------------------------------------------------
2022-05-18T04:25:49.8178754Z   test_new_subgroups_overlap_not_allowed (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2022-05-18T04:25:49.8179221Z 
2022-05-18T04:25:49.8179593Z ----------------------------------------------------------------------
2022-05-18T04:25:49.8180077Z Ran 1 test in 0.002s
2022-05-18T04:25:49.8180240Z 
2022-05-18T04:25:49.8180318Z OK (skipped=1)
2022-05-18T04:25:49.8180425Z 
2022-05-18T04:25:49.8180509Z Generating XML reports...
2022-05-18T04:25:49.8211566Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042549.xml
2022-05-18T04:25:50.6496855Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:25:50.6508000Z 
2022-05-18T04:25:50.6508434Z Running tests...
2022-05-18T04:25:50.6508853Z ----------------------------------------------------------------------
2022-05-18T04:25:50.6524660Z   test_new_subgroups_world_size_not_divisible_by_group_size (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2022-05-18T04:25:50.6525044Z 
2022-05-18T04:25:50.6525348Z ----------------------------------------------------------------------
2022-05-18T04:25:50.6525655Z Ran 1 test in 0.002s
2022-05-18T04:25:50.6525769Z 
2022-05-18T04:25:50.6525849Z OK (skipped=1)
2022-05-18T04:25:50.6525961Z 
2022-05-18T04:25:50.6526047Z Generating XML reports...
2022-05-18T04:25:50.6556510Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042550.xml
2022-05-18T04:25:51.4842319Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:25:51.4852320Z 
2022-05-18T04:25:51.4852430Z Running tests...
2022-05-18T04:25:51.4852862Z ----------------------------------------------------------------------
2022-05-18T04:25:51.7659401Z   test_output_unused_in_loss_dict_module (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19428
2022-05-18T04:25:51.7681875Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19429
2022-05-18T04:25:51.7704892Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19430
2022-05-18T04:25:52.5770889Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:25:52.5818564Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:25:52.5818964Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:25:52.5819581Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:25:52.5820127Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:25:52.5872556Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:25:52.5927360Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:25:52.5928226Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:25:52.6885050Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:25:52.8754385Z skip: Need at least 2 CUDA devices (1.390s)
2022-05-18T04:25:52.8754651Z 
2022-05-18T04:25:52.8755166Z ----------------------------------------------------------------------
2022-05-18T04:25:52.8755555Z Ran 1 test in 1.390s
2022-05-18T04:25:52.8755670Z 
2022-05-18T04:25:52.8755732Z OK (skipped=1)
2022-05-18T04:25:52.8756038Z 
2022-05-18T04:25:52.8756127Z Generating XML reports...
2022-05-18T04:25:52.8786389Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042551.xml
2022-05-18T04:25:53.7952596Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:25:53.7962697Z 
2022-05-18T04:25:53.7962832Z Running tests...
2022-05-18T04:25:53.7963371Z ----------------------------------------------------------------------
2022-05-18T04:25:54.0759254Z   test_output_unused_in_loss_tuple_module (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19481
2022-05-18T04:25:54.0781324Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19482
2022-05-18T04:25:54.0804184Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19483
2022-05-18T04:25:54.8975547Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:25:54.9076090Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:25:54.9076544Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:25:54.9077245Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:25:54.9077784Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:25:54.9078305Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:25:54.9086829Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:25:54.9087443Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:25:54.9087877Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:25:55.0856698Z skip: Need at least 2 CUDA devices (1.289s)
2022-05-18T04:25:55.0856896Z 
2022-05-18T04:25:55.0857235Z ----------------------------------------------------------------------
2022-05-18T04:25:55.0857489Z Ran 1 test in 1.289s
2022-05-18T04:25:55.0857604Z 
2022-05-18T04:25:55.0857665Z OK (skipped=1)
2022-05-18T04:25:55.0857772Z 
2022-05-18T04:25:55.0857860Z Generating XML reports...
2022-05-18T04:25:55.0887783Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042553.xml
2022-05-18T04:25:56.0142766Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:25:56.0153011Z 
2022-05-18T04:25:56.0153115Z Running tests...
2022-05-18T04:25:56.0153526Z ----------------------------------------------------------------------
2022-05-18T04:25:56.3005674Z   test_periodic_model_averager (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19534
2022-05-18T04:25:56.3031795Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19535
2022-05-18T04:25:56.3056921Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19536
2022-05-18T04:25:57.1419057Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:25:57.1461723Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:25:57.1462440Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:25:57.1463284Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:25:57.1463816Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:25:57.1521069Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:25:57.1571227Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:25:57.1571818Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:25:57.2534531Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:25:57.4107722Z skip: Need at least 2 CUDA devices (1.395s)
2022-05-18T04:25:57.4108022Z 
2022-05-18T04:25:57.4108547Z ----------------------------------------------------------------------
2022-05-18T04:25:57.4108828Z Ran 1 test in 1.395s
2022-05-18T04:25:57.4108942Z 
2022-05-18T04:25:57.4109017Z OK (skipped=1)
2022-05-18T04:25:57.4109126Z 
2022-05-18T04:25:57.4109214Z Generating XML reports...
2022-05-18T04:25:57.4141771Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042556.xml
2022-05-18T04:25:58.3390362Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:25:58.3401310Z 
2022-05-18T04:25:58.3401650Z Running tests...
2022-05-18T04:25:58.3402268Z ----------------------------------------------------------------------
2022-05-18T04:25:58.6218240Z   test_periodic_model_averager_param_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19587
2022-05-18T04:25:58.6241067Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19588
2022-05-18T04:25:58.6264220Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19589
2022-05-18T04:25:59.4881019Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:25:59.4981630Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:25:59.4982131Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:25:59.4982769Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:25:59.4983430Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:25:59.4983950Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:25:59.5089462Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:25:59.5996257Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:25:59.5996726Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:25:59.8318670Z skip: Need at least 2 CUDA devices (1.491s)
2022-05-18T04:25:59.8318993Z 
2022-05-18T04:25:59.8319359Z ----------------------------------------------------------------------
2022-05-18T04:25:59.8319663Z Ran 1 test in 1.492s
2022-05-18T04:25:59.8319777Z 
2022-05-18T04:25:59.8319863Z OK (skipped=1)
2022-05-18T04:25:59.8319973Z 
2022-05-18T04:25:59.8320047Z Generating XML reports...
2022-05-18T04:25:59.8350806Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042558.xml
2022-05-18T04:26:00.7579420Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:26:00.7589600Z 
2022-05-18T04:26:00.7589817Z Running tests...
2022-05-18T04:26:00.7590211Z ----------------------------------------------------------------------
2022-05-18T04:26:01.0326031Z   test_post_localSGD_optimizer_parity (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77123 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure IN_CI is not set and you are not using the flag --import-disabled-tests. (0.273s)
2022-05-18T04:26:01.0326748Z 
2022-05-18T04:26:01.0327025Z ----------------------------------------------------------------------
2022-05-18T04:26:01.0327269Z Ran 1 test in 0.273s
2022-05-18T04:26:01.0327369Z 
2022-05-18T04:26:01.0327443Z OK (skipped=1)
2022-05-18T04:26:01.0327552Z 
2022-05-18T04:26:01.0327637Z Generating XML reports...
2022-05-18T04:26:01.0354216Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042600.xml
2022-05-18T04:26:01.9478382Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:26:01.9488038Z 
2022-05-18T04:26:01.9488171Z Running tests...
2022-05-18T04:26:01.9488625Z ----------------------------------------------------------------------
2022-05-18T04:26:02.2216590Z   test_post_localSGD_optimizer_parity_grad_is_view (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77292 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure IN_CI is not set and you are not using the flag --import-disabled-tests. (0.272s)
2022-05-18T04:26:02.2217106Z 
2022-05-18T04:26:02.2217331Z ----------------------------------------------------------------------
2022-05-18T04:26:02.2217577Z Ran 1 test in 0.273s
2022-05-18T04:26:02.2217693Z 
2022-05-18T04:26:02.2217765Z OK (skipped=1)
2022-05-18T04:26:02.2217872Z 
2022-05-18T04:26:02.2217945Z Generating XML reports...
2022-05-18T04:26:02.2246125Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042601.xml
2022-05-18T04:26:03.1270915Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:26:03.1281732Z 
2022-05-18T04:26:03.1281889Z Running tests...
2022-05-18T04:26:03.1282285Z ----------------------------------------------------------------------
2022-05-18T04:26:03.4092974Z   test_post_localSGD_optimizer_parity_with_hierarchical_sgd (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19660
2022-05-18T04:26:03.4115437Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19661
2022-05-18T04:26:03.4138248Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19662
2022-05-18T04:26:04.2365598Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:26:04.2467075Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:26:04.2468073Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:26:04.2468734Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:26:04.2469521Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:26:04.2470050Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:26:04.2477917Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:26:04.2478990Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:26:04.2479583Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:26:04.4187402Z skip: Need at least 4 CUDA devices (1.290s)
2022-05-18T04:26:04.4187720Z 
2022-05-18T04:26:04.4188225Z ----------------------------------------------------------------------
2022-05-18T04:26:04.4188468Z Ran 1 test in 1.290s
2022-05-18T04:26:04.4188770Z 
2022-05-18T04:26:04.4188844Z OK (skipped=1)
2022-05-18T04:26:04.4188950Z 
2022-05-18T04:26:04.4189035Z Generating XML reports...
2022-05-18T04:26:04.4219163Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042603.xml
2022-05-18T04:26:05.3403189Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:26:05.3412567Z 
2022-05-18T04:26:05.3412697Z Running tests...
2022-05-18T04:26:05.3413300Z ----------------------------------------------------------------------
2022-05-18T04:26:05.6245164Z   test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19713
2022-05-18T04:26:05.6267344Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19714
2022-05-18T04:26:05.6290144Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19715
2022-05-18T04:26:06.4183829Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:26:06.4283829Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:26:06.4284431Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:26:06.4285158Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:26:06.4285704Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:26:06.4286227Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:26:06.4391393Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:26:06.5298837Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:26:06.5299232Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:26:06.7340048Z skip: Need at least 4 CUDA devices (1.392s)
2022-05-18T04:26:06.7340305Z 
2022-05-18T04:26:06.7340610Z ----------------------------------------------------------------------
2022-05-18T04:26:06.7340932Z Ran 1 test in 1.393s
2022-05-18T04:26:06.7341047Z 
2022-05-18T04:26:06.7341122Z OK (skipped=1)
2022-05-18T04:26:06.7341230Z 
2022-05-18T04:26:06.7341302Z Generating XML reports...
2022-05-18T04:26:06.7372716Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042605.xml
2022-05-18T04:26:07.6544092Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:26:07.6554145Z 
2022-05-18T04:26:07.6554270Z Running tests...
2022-05-18T04:26:07.6554720Z ----------------------------------------------------------------------
2022-05-18T04:26:07.9369017Z   test_reduce_full_group_max (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19766
2022-05-18T04:26:07.9390561Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19767
2022-05-18T04:26:07.9412850Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19768
2022-05-18T04:26:08.7272268Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:26:08.7373803Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:26:08.7374497Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:26:08.7375125Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:26:08.7375659Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:26:08.7376437Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:26:08.7383390Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:26:08.7385122Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:26:08.7385518Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:26:08.7588433Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:26:08.7592139Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:26:08.7592689Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-05-18T04:26:08.7593339Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:26:08.7593865Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:26:08.7690459Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:26:09.0462786Z ok (1.391s)
2022-05-18T04:26:09.0463167Z 
2022-05-18T04:26:09.0463639Z ----------------------------------------------------------------------
2022-05-18T04:26:09.0463885Z Ran 1 test in 1.391s
2022-05-18T04:26:09.0464001Z 
2022-05-18T04:26:09.0464064Z OK
2022-05-18T04:26:09.0464157Z 
2022-05-18T04:26:09.0464254Z Generating XML reports...
2022-05-18T04:26:09.0496719Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042607.xml
2022-05-18T04:26:09.9691138Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:26:09.9701975Z 
2022-05-18T04:26:09.9702381Z Running tests...
2022-05-18T04:26:09.9702798Z ----------------------------------------------------------------------
2022-05-18T04:26:10.2542728Z   test_reduce_full_group_min (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19831
2022-05-18T04:26:10.2565179Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19832
2022-05-18T04:26:10.2588502Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19833
2022-05-18T04:26:11.0507365Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:26:11.0508075Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:26:11.0508698Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:26:11.0509574Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:26:11.0510124Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:26:11.0510884Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:26:11.0613642Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:26:11.1520208Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:26:11.1520756Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:26:11.1828012Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:26:11.1828773Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:26:11.1829361Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-05-18T04:26:11.1830059Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:26:11.1830582Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:26:11.1831113Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:26:11.3641013Z ok (1.394s)
2022-05-18T04:26:11.3641267Z 
2022-05-18T04:26:11.3642069Z ----------------------------------------------------------------------
2022-05-18T04:26:11.3642536Z Ran 1 test in 1.394s
2022-05-18T04:26:11.3642689Z 
2022-05-18T04:26:11.3642756Z OK
2022-05-18T04:26:11.3642849Z 
2022-05-18T04:26:11.3642944Z Generating XML reports...
2022-05-18T04:26:11.3673546Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042609.xml
2022-05-18T04:26:12.2870902Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:26:12.2881665Z 
2022-05-18T04:26:12.2881940Z Running tests...
2022-05-18T04:26:12.2882559Z ----------------------------------------------------------------------
2022-05-18T04:26:12.5700329Z   test_reduce_full_group_product (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19896
2022-05-18T04:26:12.5722323Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19897
2022-05-18T04:26:12.5746191Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19898
2022-05-18T04:26:13.3719495Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:26:13.3820166Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:26:13.3820920Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:26:13.3821575Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:26:13.3822426Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:26:13.3823296Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:26:13.3830050Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:26:13.3830672Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:26:13.3831862Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:26:13.4039089Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:26:13.4039807Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:26:13.4040337Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-05-18T04:26:13.4040958Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:26:13.4041692Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:26:13.4042217Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:26:13.6797334Z ok (1.391s)
2022-05-18T04:26:13.6797607Z 
2022-05-18T04:26:13.6798143Z ----------------------------------------------------------------------
2022-05-18T04:26:13.6798770Z Ran 1 test in 1.391s
2022-05-18T04:26:13.6798890Z 
2022-05-18T04:26:13.6798952Z OK
2022-05-18T04:26:13.6799032Z 
2022-05-18T04:26:13.6799130Z Generating XML reports...
2022-05-18T04:26:13.6829122Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042612.xml
2022-05-18T04:26:14.6000042Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:26:14.6010120Z 
2022-05-18T04:26:14.6010216Z Running tests...
2022-05-18T04:26:14.6010914Z ----------------------------------------------------------------------
2022-05-18T04:26:14.8824331Z   test_reduce_full_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19961
2022-05-18T04:26:14.8847433Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19962
2022-05-18T04:26:14.8870527Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19963
2022-05-18T04:26:15.6979363Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:26:15.7040118Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:26:15.7040949Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:26:15.7041460Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:26:15.7041977Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:26:15.7080921Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:26:15.7152135Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:26:15.8091393Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:26:15.8151399Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:26:15.8398435Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:26:15.8499667Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-05-18T04:26:15.8500316Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:26:15.8501370Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:26:15.8502139Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:26:15.8502669Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:26:16.1924254Z ok (1.591s)
2022-05-18T04:26:16.1924485Z 
2022-05-18T04:26:16.1924940Z ----------------------------------------------------------------------
2022-05-18T04:26:16.1925333Z Ran 1 test in 1.591s
2022-05-18T04:26:16.1925519Z 
2022-05-18T04:26:16.1925616Z OK
2022-05-18T04:26:16.1925768Z 
2022-05-18T04:26:16.1925918Z Generating XML reports...
2022-05-18T04:26:16.1957351Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042614.xml
2022-05-18T04:26:17.1172918Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:26:17.1182805Z 
2022-05-18T04:26:17.1183088Z Running tests...
2022-05-18T04:26:17.1183724Z ----------------------------------------------------------------------
2022-05-18T04:26:17.4018269Z   test_reduce_group_max (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20026
2022-05-18T04:26:17.4040482Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20027
2022-05-18T04:26:17.4063322Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20028
2022-05-18T04:26:18.2013273Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:26:18.2013927Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:26:18.2014492Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:26:18.2015427Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:26:18.2016306Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:26:18.2017197Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:26:18.2122060Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:26:18.2122645Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:26:18.2227019Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:26:18.2227428Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-05-18T04:26:18.3025697Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:26:18.3027852Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:26:18.3028546Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:26:18.3036078Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:26:18.3036700Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:26:18.6115459Z ok (1.493s)
2022-05-18T04:26:18.6115692Z 
2022-05-18T04:26:18.6116058Z ----------------------------------------------------------------------
2022-05-18T04:26:18.6116342Z Ran 1 test in 1.493s
2022-05-18T04:26:18.6116524Z 
2022-05-18T04:26:18.6116589Z OK
2022-05-18T04:26:18.6116669Z 
2022-05-18T04:26:18.6116765Z Generating XML reports...
2022-05-18T04:26:18.6146938Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042617.xml
2022-05-18T04:26:19.5370529Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:26:19.5380538Z 
2022-05-18T04:26:19.5380658Z Running tests...
2022-05-18T04:26:19.5381028Z ----------------------------------------------------------------------
2022-05-18T04:26:19.8170121Z   test_reduce_group_min (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20087
2022-05-18T04:26:19.8192563Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20088
2022-05-18T04:26:19.8215181Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20089
2022-05-18T04:26:20.6246338Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:26:20.6346171Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:26:20.6346896Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:26:20.6347799Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:26:20.6348333Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:26:20.6349030Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:26:20.6453666Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:26:20.7358948Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:26:20.7359379Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:26:20.7360569Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:26:20.7564067Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-05-18T04:26:20.7564494Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:26:20.7565141Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:26:20.7565904Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:26:20.7665276Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:26:21.0268007Z ok (1.488s)
2022-05-18T04:26:21.0268275Z 
2022-05-18T04:26:21.0268676Z ----------------------------------------------------------------------
2022-05-18T04:26:21.0268930Z Ran 1 test in 1.489s
2022-05-18T04:26:21.0269046Z 
2022-05-18T04:26:21.0269110Z OK
2022-05-18T04:26:21.0269202Z 
2022-05-18T04:26:21.0269294Z Generating XML reports...
2022-05-18T04:26:21.0298587Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042619.xml
2022-05-18T04:26:21.9437875Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:26:21.9447583Z 
2022-05-18T04:26:21.9447682Z Running tests...
2022-05-18T04:26:21.9448091Z ----------------------------------------------------------------------
2022-05-18T04:26:22.2246446Z   test_reduce_group_product (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20148
2022-05-18T04:26:22.2269242Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20149
2022-05-18T04:26:22.2291993Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20150
2022-05-18T04:26:23.0452276Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:26:23.0452794Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:26:23.0453169Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:26:23.0453775Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:26:23.0454328Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:26:23.0454853Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:26:23.0558637Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:26:23.1463556Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:26:23.1463980Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:26:23.1464611Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:26:23.1569452Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-05-18T04:26:23.1570106Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:26:23.1570811Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:26:23.1571335Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:26:23.1668261Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:26:23.5347963Z ok (1.590s)
2022-05-18T04:26:23.5348238Z 
2022-05-18T04:26:23.5348761Z ----------------------------------------------------------------------
2022-05-18T04:26:23.5349052Z Ran 1 test in 1.590s
2022-05-18T04:26:23.5349167Z 
2022-05-18T04:26:23.5349229Z OK
2022-05-18T04:26:23.5349328Z 
2022-05-18T04:26:23.5349422Z Generating XML reports...
2022-05-18T04:26:23.5379127Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042621.xml
2022-05-18T04:26:24.4518659Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:26:24.4527576Z 
2022-05-18T04:26:24.4527673Z Running tests...
2022-05-18T04:26:24.4528445Z ----------------------------------------------------------------------
2022-05-18T04:26:24.7322963Z   test_reduce_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20209
2022-05-18T04:26:24.7345625Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20210
2022-05-18T04:26:24.7368263Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20211
2022-05-18T04:26:25.6096243Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:26:25.6196732Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:26:25.6197501Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:26:25.6198347Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:26:25.6198879Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:26:25.6199387Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:26:25.6305910Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:26:25.6306689Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:26:25.6511549Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-05-18T04:26:25.6512202Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:26:25.7208687Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:26:25.7211000Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:26:25.7211659Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:26:25.7219976Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:26:25.7220505Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:26:26.0421292Z ok (1.589s)
2022-05-18T04:26:26.0421451Z 
2022-05-18T04:26:26.0421853Z ----------------------------------------------------------------------
2022-05-18T04:26:26.0423595Z Ran 1 test in 1.589s
2022-05-18T04:26:26.0423828Z 
2022-05-18T04:26:26.0424084Z OK
2022-05-18T04:26:26.0424185Z 
2022-05-18T04:26:26.0424289Z Generating XML reports...
2022-05-18T04:26:26.0453843Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042624.xml
2022-05-18T04:26:26.9656523Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:26:26.9666740Z 
2022-05-18T04:26:26.9666924Z Running tests...
2022-05-18T04:26:26.9667276Z ----------------------------------------------------------------------
2022-05-18T04:26:27.2469453Z   test_reduce_max (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20270
2022-05-18T04:26:27.2492332Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20271
2022-05-18T04:26:27.2515923Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20272
2022-05-18T04:26:28.1031404Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:26:28.1032166Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:26:28.1032851Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:26:28.1033655Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:26:28.1034185Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:26:28.1034693Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:26:28.1137979Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:26:28.2042282Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:26:28.2042993Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:26:28.4569426Z ok (1.490s)
2022-05-18T04:26:28.4569665Z 
2022-05-18T04:26:28.4570103Z ----------------------------------------------------------------------
2022-05-18T04:26:28.4570371Z Ran 1 test in 1.490s
2022-05-18T04:26:28.4570487Z 
2022-05-18T04:26:28.4570549Z OK
2022-05-18T04:26:28.4570642Z 
2022-05-18T04:26:28.4570721Z Generating XML reports...
2022-05-18T04:26:28.4601376Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042626.xml
2022-05-18T04:26:29.3859750Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:26:29.3869918Z 
2022-05-18T04:26:29.3870060Z Running tests...
2022-05-18T04:26:29.3870488Z ----------------------------------------------------------------------
2022-05-18T04:26:29.6671633Z   test_reduce_min (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20326
2022-05-18T04:26:29.6693473Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20327
2022-05-18T04:26:29.6717187Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20328
2022-05-18T04:26:30.4906478Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:26:30.5008118Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:26:30.5009071Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:26:30.5010256Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:26:30.5011092Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:26:30.5011847Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:26:30.5019318Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:26:30.5020193Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:26:30.5022157Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:26:30.7768265Z ok (1.390s)
2022-05-18T04:26:30.7768450Z 
2022-05-18T04:26:30.7768825Z ----------------------------------------------------------------------
2022-05-18T04:26:30.7769095Z Ran 1 test in 1.390s
2022-05-18T04:26:30.7769275Z 
2022-05-18T04:26:30.7769341Z OK
2022-05-18T04:26:30.7769433Z 
2022-05-18T04:26:30.7769511Z Generating XML reports...
2022-05-18T04:26:30.7801492Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042629.xml
2022-05-18T04:26:31.6991841Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:26:31.7001753Z 
2022-05-18T04:26:31.7001888Z Running tests...
2022-05-18T04:26:31.7002369Z ----------------------------------------------------------------------
2022-05-18T04:26:31.7020859Z   test_reduce_multigpu (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl backend supports reduce multigpu (0.002s)
2022-05-18T04:26:31.7021371Z 
2022-05-18T04:26:31.7021767Z ----------------------------------------------------------------------
2022-05-18T04:26:31.7022014Z Ran 1 test in 0.002s
2022-05-18T04:26:31.7022135Z 
2022-05-18T04:26:31.7022197Z OK (skipped=1)
2022-05-18T04:26:31.7022304Z 
2022-05-18T04:26:31.7022390Z Generating XML reports...
2022-05-18T04:26:31.7053387Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042631.xml
2022-05-18T04:26:32.5328048Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:26:32.5338132Z 
2022-05-18T04:26:32.5338277Z Running tests...
2022-05-18T04:26:32.5338881Z ----------------------------------------------------------------------
2022-05-18T04:26:32.8135262Z   test_reduce_product (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20392
2022-05-18T04:26:32.8157630Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20393
2022-05-18T04:26:32.8180683Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20394
2022-05-18T04:26:33.6153659Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:26:33.6255778Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:26:33.6256188Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:26:33.6256800Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:26:33.6257356Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:26:33.6257881Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:26:33.6265497Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:26:33.6266182Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:26:33.6266701Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:26:33.9231149Z ok (1.389s)
2022-05-18T04:26:33.9231415Z 
2022-05-18T04:26:33.9231941Z ----------------------------------------------------------------------
2022-05-18T04:26:33.9232222Z Ran 1 test in 1.389s
2022-05-18T04:26:33.9232325Z 
2022-05-18T04:26:33.9232593Z OK
2022-05-18T04:26:33.9232686Z 
2022-05-18T04:26:33.9232782Z Generating XML reports...
2022-05-18T04:26:33.9263155Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042632.xml
2022-05-18T04:26:34.8418666Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:26:34.8428887Z 
2022-05-18T04:26:34.8429065Z Running tests...
2022-05-18T04:26:34.8429408Z ----------------------------------------------------------------------
2022-05-18T04:26:35.1250529Z   test_reduce_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20448
2022-05-18T04:26:35.1272804Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20449
2022-05-18T04:26:35.1296257Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20450
2022-05-18T04:26:35.9325052Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:26:35.9325681Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:26:35.9326060Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:26:35.9326669Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:26:35.9327192Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:26:35.9327717Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:26:35.9432893Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:26:36.0338288Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:26:36.0338850Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:26:36.3350687Z ok (1.492s)
2022-05-18T04:26:36.3350952Z 
2022-05-18T04:26:36.3351486Z ----------------------------------------------------------------------
2022-05-18T04:26:36.3351896Z Ran 1 test in 1.492s
2022-05-18T04:26:36.3352015Z 
2022-05-18T04:26:36.3352078Z OK
2022-05-18T04:26:36.3352175Z 
2022-05-18T04:26:36.3352261Z Generating XML reports...
2022-05-18T04:26:36.3383905Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042634.xml
2022-05-18T04:26:37.2661009Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:26:37.2671451Z 
2022-05-18T04:26:37.2671577Z Running tests...
2022-05-18T04:26:37.2672169Z ----------------------------------------------------------------------
2022-05-18T04:26:37.2690068Z   test_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA reduce (0.002s)
2022-05-18T04:26:37.2690521Z 
2022-05-18T04:26:37.2690821Z ----------------------------------------------------------------------
2022-05-18T04:26:37.2691170Z Ran 1 test in 0.002s
2022-05-18T04:26:37.2691352Z 
2022-05-18T04:26:37.2691468Z OK (skipped=1)
2022-05-18T04:26:37.2691565Z 
2022-05-18T04:26:37.2691657Z Generating XML reports...
2022-05-18T04:26:37.2724412Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042637.xml
2022-05-18T04:26:38.1067860Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:26:38.1078498Z 
2022-05-18T04:26:38.1078627Z Running tests...
2022-05-18T04:26:38.1079218Z ----------------------------------------------------------------------
2022-05-18T04:26:38.1097692Z   test_reduce_sum_cuda_twice (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA reduce (0.002s)
2022-05-18T04:26:38.1098116Z 
2022-05-18T04:26:38.1098795Z ----------------------------------------------------------------------
2022-05-18T04:26:38.1099198Z Ran 1 test in 0.002s
2022-05-18T04:26:38.1099377Z 
2022-05-18T04:26:38.1099479Z OK (skipped=1)
2022-05-18T04:26:38.1099653Z 
2022-05-18T04:26:38.1099920Z Generating XML reports...
2022-05-18T04:26:38.1131616Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042638.xml
2022-05-18T04:26:38.9381941Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:26:38.9391304Z 
2022-05-18T04:26:38.9391488Z Running tests...
2022-05-18T04:26:38.9391890Z ----------------------------------------------------------------------
2022-05-18T04:26:39.2180193Z   test_reduce_sum_twice (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20524
2022-05-18T04:26:39.2201846Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20525
2022-05-18T04:26:39.2225559Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20526
2022-05-18T04:26:40.0120149Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:26:40.0188257Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:26:40.0188996Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:26:40.0189698Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:26:40.0190216Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:26:40.0221508Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:26:40.0297113Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:26:40.0297994Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:26:40.1234083Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:26:40.4279438Z ok (1.488s)
2022-05-18T04:26:40.4279692Z 
2022-05-18T04:26:40.4280215Z ----------------------------------------------------------------------
2022-05-18T04:26:40.4280590Z Ran 1 test in 1.489s
2022-05-18T04:26:40.4280706Z 
2022-05-18T04:26:40.4280770Z OK
2022-05-18T04:26:40.4280863Z 
2022-05-18T04:26:40.4280957Z Generating XML reports...
2022-05-18T04:26:40.4317929Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042638.xml
2022-05-18T04:26:41.3602949Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:26:41.3613290Z 
2022-05-18T04:26:41.3613382Z Running tests...
2022-05-18T04:26:41.3614153Z ----------------------------------------------------------------------
2022-05-18T04:26:41.6431442Z   test_scatter (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20580
2022-05-18T04:26:41.6453247Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20581
2022-05-18T04:26:41.6476253Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20582
2022-05-18T04:26:42.4506915Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:26:42.4507575Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:26:42.4507947Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:26:42.4508567Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:26:42.4509323Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:26:42.4510273Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:26:42.4516998Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:26:42.4517758Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:26:42.4518500Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:26:42.7527903Z ok (1.391s)
2022-05-18T04:26:42.7528102Z 
2022-05-18T04:26:42.7528594Z ----------------------------------------------------------------------
2022-05-18T04:26:42.7529052Z Ran 1 test in 1.391s
2022-05-18T04:26:42.7529241Z 
2022-05-18T04:26:42.7529339Z OK
2022-05-18T04:26:42.7529519Z 
2022-05-18T04:26:42.7529695Z Generating XML reports...
2022-05-18T04:26:42.7560770Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042641.xml
2022-05-18T04:26:43.6730941Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:26:43.6741347Z 
2022-05-18T04:26:43.6741651Z Running tests...
2022-05-18T04:26:43.6742108Z ----------------------------------------------------------------------
2022-05-18T04:26:43.9554062Z   test_scatter_checks (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20636
2022-05-18T04:26:43.9576424Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20637
2022-05-18T04:26:43.9598674Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20638
2022-05-18T04:26:44.7626403Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:26:44.7726890Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:26:44.7727526Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:26:44.7728526Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:26:44.7729411Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:26:44.7729957Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:26:44.7834313Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:26:44.8740305Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:26:44.8740911Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:26:45.0650136Z ok (1.391s)
2022-05-18T04:26:45.0650363Z 
2022-05-18T04:26:45.0650901Z ----------------------------------------------------------------------
2022-05-18T04:26:45.0651266Z Ran 1 test in 1.391s
2022-05-18T04:26:45.0651389Z 
2022-05-18T04:26:45.0651452Z OK
2022-05-18T04:26:45.0651544Z 
2022-05-18T04:26:45.0651639Z Generating XML reports...
2022-05-18T04:26:45.0682282Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042643.xml
2022-05-18T04:26:45.9838832Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:26:45.9848070Z 
2022-05-18T04:26:45.9848213Z Running tests...
2022-05-18T04:26:45.9848807Z ----------------------------------------------------------------------
2022-05-18T04:26:46.2659117Z   test_scatter_complex (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20689
2022-05-18T04:26:46.2681723Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20690
2022-05-18T04:26:46.2705232Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20691
2022-05-18T04:26:47.0661911Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:26:47.0761780Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:26:47.0762338Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:26:47.0763198Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:26:47.0763775Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:26:47.0764293Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:26:47.0869998Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:26:47.1777194Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:26:47.1777759Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:26:47.4756855Z ok (1.491s)
2022-05-18T04:26:47.4757057Z 
2022-05-18T04:26:47.4757400Z ----------------------------------------------------------------------
2022-05-18T04:26:47.4757680Z Ran 1 test in 1.491s
2022-05-18T04:26:47.4757811Z 
2022-05-18T04:26:47.4757872Z OK
2022-05-18T04:26:47.4757968Z 
2022-05-18T04:26:47.4758062Z Generating XML reports...
2022-05-18T04:26:47.4788846Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042645.xml
2022-05-18T04:26:48.3902526Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:26:48.3913170Z 
2022-05-18T04:26:48.3913277Z Running tests...
2022-05-18T04:26:48.3913765Z ----------------------------------------------------------------------
2022-05-18T04:26:48.3929096Z   test_scatter_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA gather (0.001s)
2022-05-18T04:26:48.3929330Z 
2022-05-18T04:26:48.3929744Z ----------------------------------------------------------------------
2022-05-18T04:26:48.3930189Z Ran 1 test in 0.002s
2022-05-18T04:26:48.3930400Z 
2022-05-18T04:26:48.3930492Z OK (skipped=1)
2022-05-18T04:26:48.3930599Z 
2022-05-18T04:26:48.3930671Z Generating XML reports...
2022-05-18T04:26:48.3961515Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042648.xml
2022-05-18T04:26:49.2224684Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:26:49.2235616Z 
2022-05-18T04:26:49.2235926Z Running tests...
2022-05-18T04:26:49.2236620Z ----------------------------------------------------------------------
2022-05-18T04:26:49.2252089Z   test_scatter_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA gather (0.002s)
2022-05-18T04:26:49.2252500Z 
2022-05-18T04:26:49.2252956Z ----------------------------------------------------------------------
2022-05-18T04:26:49.2253413Z Ran 1 test in 0.002s
2022-05-18T04:26:49.2253549Z 
2022-05-18T04:26:49.2253628Z OK (skipped=1)
2022-05-18T04:26:49.2253736Z 
2022-05-18T04:26:49.2253822Z Generating XML reports...
2022-05-18T04:26:49.2285953Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042649.xml
2022-05-18T04:26:50.0599923Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:26:50.0610828Z 
2022-05-18T04:26:50.0611169Z Running tests...
2022-05-18T04:26:50.0611597Z ----------------------------------------------------------------------
2022-05-18T04:26:50.3412181Z   test_scatter_full_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20765
2022-05-18T04:26:50.3434881Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20766
2022-05-18T04:26:50.3457929Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20767
2022-05-18T04:26:51.1521434Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:26:51.1621645Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:26:51.1622229Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:26:51.1623316Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:26:51.1624194Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:26:51.1625023Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:26:51.1728986Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:26:51.2635245Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:26:51.2635727Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:26:51.2841243Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:26:51.2941471Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:26:51.2941958Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-05-18T04:26:51.2943181Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:26:51.2943795Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:26:51.2944322Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:26:51.5510022Z ok (1.490s)
2022-05-18T04:26:51.5510252Z 
2022-05-18T04:26:51.5510816Z ----------------------------------------------------------------------
2022-05-18T04:26:51.5511172Z Ran 1 test in 1.490s
2022-05-18T04:26:51.5511289Z 
2022-05-18T04:26:51.5511351Z OK
2022-05-18T04:26:51.5511449Z 
2022-05-18T04:26:51.5511544Z Generating XML reports...
2022-05-18T04:26:51.5543561Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042650.xml
2022-05-18T04:26:52.4750173Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:26:52.4760219Z 
2022-05-18T04:26:52.4760445Z Running tests...
2022-05-18T04:26:52.4761221Z ----------------------------------------------------------------------
2022-05-18T04:26:52.7597568Z   test_scatter_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20830
2022-05-18T04:26:52.7620068Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20831
2022-05-18T04:26:52.7642965Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20832
2022-05-18T04:26:53.5709461Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:26:53.5810153Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:26:53.5810756Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:26:53.5811777Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:26:53.5812799Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:26:53.5813421Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:26:53.5917844Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:26:53.6822396Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:26:53.6823059Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:26:53.6824335Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:26:53.7029136Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:26:53.7029730Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-05-18T04:26:53.7030710Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:26:53.7031407Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:26:53.7129424Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:26:54.0698883Z ok (1.594s)
2022-05-18T04:26:54.0699119Z 
2022-05-18T04:26:54.0699549Z ----------------------------------------------------------------------
2022-05-18T04:26:54.0699951Z Ran 1 test in 1.594s
2022-05-18T04:26:54.0700148Z 
2022-05-18T04:26:54.0700232Z OK
2022-05-18T04:26:54.0700371Z 
2022-05-18T04:26:54.0700514Z Generating XML reports...
2022-05-18T04:26:54.0733043Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042652.xml
2022-05-18T04:26:54.9877667Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:26:54.9887199Z 
2022-05-18T04:26:54.9887348Z Running tests...
2022-05-18T04:26:54.9887984Z ----------------------------------------------------------------------
2022-05-18T04:26:55.2705634Z   test_scatter_object_list (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20891
2022-05-18T04:26:55.2728306Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20892
2022-05-18T04:26:55.2750914Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20893
2022-05-18T04:26:56.0893591Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:26:56.0993775Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:26:56.0994280Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:26:56.0995037Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:26:56.0995569Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:26:56.0996148Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:26:56.1101786Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:26:56.2008737Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:26:56.2009167Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:26:56.3800321Z ok (1.391s)
2022-05-18T04:26:56.3800526Z 
2022-05-18T04:26:56.3801012Z ----------------------------------------------------------------------
2022-05-18T04:26:56.3801673Z Ran 1 test in 1.391s
2022-05-18T04:26:56.3801788Z 
2022-05-18T04:26:56.3801851Z OK
2022-05-18T04:26:56.3801930Z 
2022-05-18T04:26:56.3802100Z Generating XML reports...
2022-05-18T04:26:56.3833537Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042654.xml
2022-05-18T04:26:57.3001911Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:26:57.3012368Z 
2022-05-18T04:26:57.3012501Z Running tests...
2022-05-18T04:26:57.3013081Z ----------------------------------------------------------------------
2022-05-18T04:26:57.5824965Z   test_send_recv (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20944
2022-05-18T04:26:57.5846343Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20945
2022-05-18T04:26:57.5868877Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20946
2022-05-18T04:26:58.3912324Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:26:58.4013573Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:26:58.4013976Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:26:58.4014600Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:26:58.4015133Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:26:58.4015657Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:26:58.4121123Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:26:58.5027172Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:26:58.5027564Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:26:58.6920215Z ok (1.390s)
2022-05-18T04:26:58.6920456Z 
2022-05-18T04:26:58.6920882Z ----------------------------------------------------------------------
2022-05-18T04:26:58.6921345Z Ran 1 test in 1.391s
2022-05-18T04:26:58.6921556Z 
2022-05-18T04:26:58.6921671Z OK
2022-05-18T04:26:58.6921806Z 
2022-05-18T04:26:58.6921885Z Generating XML reports...
2022-05-18T04:26:58.6952229Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042657.xml
2022-05-18T04:26:59.6079132Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:26:59.6088952Z 
2022-05-18T04:26:59.6089113Z Running tests...
2022-05-18T04:26:59.6089738Z ----------------------------------------------------------------------
2022-05-18T04:26:59.8897078Z   test_send_recv_any_source (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20997
2022-05-18T04:26:59.8918939Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20998
2022-05-18T04:26:59.8940954Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20999
2022-05-18T04:27:00.7214045Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:27:00.7214697Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:27:00.7215314Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:27:00.7216265Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:27:00.7216789Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:27:00.7217854Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:27:00.7320649Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:27:00.8227224Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:27:00.8227945Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:27:00.9991587Z ok (1.390s)
2022-05-18T04:27:00.9991824Z 
2022-05-18T04:27:00.9992294Z ----------------------------------------------------------------------
2022-05-18T04:27:00.9992693Z Ran 1 test in 1.390s
2022-05-18T04:27:00.9992853Z 
2022-05-18T04:27:00.9992958Z OK
2022-05-18T04:27:00.9993099Z 
2022-05-18T04:27:00.9993239Z Generating XML reports...
2022-05-18T04:27:01.0024672Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042659.xml
2022-05-18T04:27:01.9336769Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:27:01.9363609Z 
2022-05-18T04:27:01.9363955Z Running tests...
2022-05-18T04:27:01.9364673Z ----------------------------------------------------------------------
2022-05-18T04:27:02.2205635Z   test_send_recv_any_source_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21050
2022-05-18T04:27:02.2228739Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21051
2022-05-18T04:27:02.2252194Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21052
2022-05-18T04:27:03.0648250Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:27:03.0648862Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:27:03.0649508Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:27:03.0650474Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:27:03.0651369Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:27:03.0652027Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:27:03.0756041Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:27:03.0756863Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:27:03.1660340Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:27:03.4303680Z ok (1.494s)
2022-05-18T04:27:03.4303931Z 
2022-05-18T04:27:03.4304390Z ----------------------------------------------------------------------
2022-05-18T04:27:03.4304792Z Ran 1 test in 1.494s
2022-05-18T04:27:03.4304999Z 
2022-05-18T04:27:03.4305093Z OK
2022-05-18T04:27:03.4305226Z 
2022-05-18T04:27:03.4305368Z Generating XML reports...
2022-05-18T04:27:03.4336219Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042701.xml
2022-05-18T04:27:04.3528554Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:27:04.3539349Z 
2022-05-18T04:27:04.3539742Z Running tests...
2022-05-18T04:27:04.3540182Z ----------------------------------------------------------------------
2022-05-18T04:27:04.6356035Z   test_send_recv_any_source_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21106
2022-05-18T04:27:04.6378026Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21107
2022-05-18T04:27:04.6400684Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21108
2022-05-18T04:27:05.4788022Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:27:05.4887377Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:27:05.4887869Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:27:05.4888752Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:27:05.4889291Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:27:05.4889821Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:27:05.4997981Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:27:05.4998370Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:27:05.5901238Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:27:05.8453326Z ok (1.491s)
2022-05-18T04:27:05.8453534Z 
2022-05-18T04:27:05.8454040Z ----------------------------------------------------------------------
2022-05-18T04:27:05.8454481Z Ran 1 test in 1.491s
2022-05-18T04:27:05.8454599Z 
2022-05-18T04:27:05.8454663Z OK
2022-05-18T04:27:05.8454758Z 
2022-05-18T04:27:05.8454839Z Generating XML reports...
2022-05-18T04:27:05.8485022Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042704.xml
2022-05-18T04:27:06.7675111Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:27:06.7685696Z 
2022-05-18T04:27:06.7686004Z Running tests...
2022-05-18T04:27:06.7686640Z ----------------------------------------------------------------------
2022-05-18T04:27:07.0498592Z   test_send_recv_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21162
2022-05-18T04:27:07.0521202Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21163
2022-05-18T04:27:07.0543512Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21164
2022-05-18T04:27:07.8592700Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:27:07.8593225Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:27:07.8593861Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:27:07.8594326Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:27:07.8595049Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:27:07.8595934Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:27:07.8698942Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:27:07.9607992Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:27:07.9608523Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:27:08.1593674Z ok (1.391s)
2022-05-18T04:27:08.1593910Z 
2022-05-18T04:27:08.1594435Z ----------------------------------------------------------------------
2022-05-18T04:27:08.1594834Z Ran 1 test in 1.391s
2022-05-18T04:27:08.1594952Z 
2022-05-18T04:27:08.1595229Z OK
2022-05-18T04:27:08.1595326Z 
2022-05-18T04:27:08.1595422Z Generating XML reports...
2022-05-18T04:27:08.1625876Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042706.xml
2022-05-18T04:27:09.0887957Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:27:09.0898487Z 
2022-05-18T04:27:09.0898623Z Running tests...
2022-05-18T04:27:09.0899206Z ----------------------------------------------------------------------
2022-05-18T04:27:09.0915632Z   test_send_recv_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Send Recv Only (0.001s)
2022-05-18T04:27:09.0916118Z 
2022-05-18T04:27:09.0916450Z ----------------------------------------------------------------------
2022-05-18T04:27:09.0916701Z Ran 1 test in 0.002s
2022-05-18T04:27:09.0916818Z 
2022-05-18T04:27:09.0916897Z OK (skipped=1)
2022-05-18T04:27:09.0917006Z 
2022-05-18T04:27:09.0917091Z Generating XML reports...
2022-05-18T04:27:09.0947113Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042709.xml
2022-05-18T04:27:09.9245358Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:27:09.9254624Z 
2022-05-18T04:27:09.9254722Z Running tests...
2022-05-18T04:27:09.9255213Z ----------------------------------------------------------------------
2022-05-18T04:27:09.9271407Z   test_send_recv_nccl_autograd_profiler (__main__.TestDistBackendWithSpawn) ... skip: NCCL Send Recv Only (0.001s)
2022-05-18T04:27:09.9271848Z 
2022-05-18T04:27:09.9272309Z ----------------------------------------------------------------------
2022-05-18T04:27:09.9272571Z Ran 1 test in 0.002s
2022-05-18T04:27:09.9272696Z 
2022-05-18T04:27:09.9272771Z OK (skipped=1)
2022-05-18T04:27:09.9272868Z 
2022-05-18T04:27:09.9272958Z Generating XML reports...
2022-05-18T04:27:09.9309344Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042709.xml
2022-05-18T04:27:10.7602638Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:27:10.7612948Z 
2022-05-18T04:27:10.7613285Z Running tests...
2022-05-18T04:27:10.7613896Z ----------------------------------------------------------------------
2022-05-18T04:27:10.7631506Z   test_send_recv_nccl_torch_profiler (__main__.TestDistBackendWithSpawn) ... skip: NCCL Send Recv Only (0.002s)
2022-05-18T04:27:10.7631842Z 
2022-05-18T04:27:10.7632108Z ----------------------------------------------------------------------
2022-05-18T04:27:10.7632439Z Ran 1 test in 0.002s
2022-05-18T04:27:10.7632569Z 
2022-05-18T04:27:10.7632661Z OK (skipped=1)
2022-05-18T04:27:10.7632772Z 
2022-05-18T04:27:10.7632859Z Generating XML reports...
2022-05-18T04:27:10.7663424Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042710.xml
2022-05-18T04:27:11.5973637Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:27:11.5984452Z 
2022-05-18T04:27:11.5984988Z Running tests...
2022-05-18T04:27:11.5985411Z ----------------------------------------------------------------------
2022-05-18T04:27:11.8798317Z   test_send_recv_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21248
2022-05-18T04:27:11.8822002Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21249
2022-05-18T04:27:11.8845067Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21250
2022-05-18T04:27:12.6712078Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:27:12.6812633Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:27:12.6813200Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:27:12.6814001Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:27:12.6814596Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:27:12.6815149Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:27:12.6822860Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:27:12.6823511Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:27:12.6825135Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:27:12.8894011Z ok (1.291s)
2022-05-18T04:27:12.8894302Z 
2022-05-18T04:27:12.8894794Z ----------------------------------------------------------------------
2022-05-18T04:27:12.8895078Z Ran 1 test in 1.291s
2022-05-18T04:27:12.8895181Z 
2022-05-18T04:27:12.8895242Z OK
2022-05-18T04:27:12.8895335Z 
2022-05-18T04:27:12.8895430Z Generating XML reports...
2022-05-18T04:27:12.8925752Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042711.xml
2022-05-18T04:27:13.8107178Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:27:13.8118009Z 
2022-05-18T04:27:13.8118313Z Running tests...
2022-05-18T04:27:13.8118926Z ----------------------------------------------------------------------
2022-05-18T04:27:14.0931575Z   test_send_recv_with_tag (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21304
2022-05-18T04:27:14.0953661Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21305
2022-05-18T04:27:14.0976204Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21306
2022-05-18T04:27:14.9204383Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:27:14.9305098Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:27:14.9305570Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:27:14.9306192Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:27:14.9306720Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:27:14.9307241Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:27:14.9414526Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:27:14.9415141Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:27:15.0318587Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:27:15.2027438Z ok (1.391s)
2022-05-18T04:27:15.2027699Z 
2022-05-18T04:27:15.2028047Z ----------------------------------------------------------------------
2022-05-18T04:27:15.2028297Z Ran 1 test in 1.391s
2022-05-18T04:27:15.2028415Z 
2022-05-18T04:27:15.2028477Z OK
2022-05-18T04:27:15.2028567Z 
2022-05-18T04:27:15.2028661Z Generating XML reports...
2022-05-18T04:27:15.2058248Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042713.xml
2022-05-18T04:27:16.1233092Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:27:16.1243950Z 
2022-05-18T04:27:16.1244367Z Running tests...
2022-05-18T04:27:16.1244753Z ----------------------------------------------------------------------
2022-05-18T04:27:16.4059015Z   test_send_recv_with_tag_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21357
2022-05-18T04:27:16.4081527Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21358
2022-05-18T04:27:16.4103747Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21359
2022-05-18T04:27:17.2293073Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:27:17.2293647Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:27:17.2294575Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:27:17.2295237Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:27:17.2296022Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:27:17.2296794Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:27:17.2303239Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:27:17.2304273Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:27:17.2304810Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:27:17.5156048Z ok (1.391s)
2022-05-18T04:27:17.5156276Z 
2022-05-18T04:27:17.5156789Z ----------------------------------------------------------------------
2022-05-18T04:27:17.5157179Z Ran 1 test in 1.391s
2022-05-18T04:27:17.5157298Z 
2022-05-18T04:27:17.5157361Z OK
2022-05-18T04:27:17.5157453Z 
2022-05-18T04:27:17.5157548Z Generating XML reports...
2022-05-18T04:27:17.5187500Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042716.xml
2022-05-18T04:27:18.4410980Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:27:18.4420892Z 
2022-05-18T04:27:18.4421028Z Running tests...
2022-05-18T04:27:18.4421418Z ----------------------------------------------------------------------
2022-05-18T04:27:18.7248856Z   test_send_recv_with_tag_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21413
2022-05-18T04:27:18.7271886Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21414
2022-05-18T04:27:18.7294386Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21415
2022-05-18T04:27:19.4928024Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:27:19.4988147Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:27:19.4988566Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:27:19.4989208Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:27:19.4989734Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:27:19.5029879Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:27:19.5096939Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:27:19.5097813Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:27:19.6040005Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:27:19.8346751Z ok (1.392s)
2022-05-18T04:27:19.8347191Z 
2022-05-18T04:27:19.8347703Z ----------------------------------------------------------------------
2022-05-18T04:27:19.8348167Z Ran 1 test in 1.392s
2022-05-18T04:27:19.8348313Z 
2022-05-18T04:27:19.8348452Z OK
2022-05-18T04:27:19.8348545Z 
2022-05-18T04:27:19.8348625Z Generating XML reports...
2022-05-18T04:27:19.8378505Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042718.xml
2022-05-18T04:27:20.7491608Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:27:20.7501061Z 
2022-05-18T04:27:20.7501179Z Running tests...
2022-05-18T04:27:20.7501594Z ----------------------------------------------------------------------
2022-05-18T04:27:21.0306985Z   test_sparse_all_reduce_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21469
2022-05-18T04:27:21.0330844Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21470
2022-05-18T04:27:21.0353805Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21471
2022-05-18T04:27:21.8225063Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:27:21.8316076Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:27:21.8316479Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:27:21.8317090Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:27:21.8317621Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:27:21.8326079Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:27:21.8425254Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:27:21.8425812Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:27:21.9336994Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:27:22.1405946Z ok (1.390s)
2022-05-18T04:27:22.1406187Z 
2022-05-18T04:27:22.1406529Z ----------------------------------------------------------------------
2022-05-18T04:27:22.1406829Z Ran 1 test in 1.390s
2022-05-18T04:27:22.1406947Z 
2022-05-18T04:27:22.1407010Z OK
2022-05-18T04:27:22.1407105Z 
2022-05-18T04:27:22.1407200Z Generating XML reports...
2022-05-18T04:27:22.1437713Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042720.xml
2022-05-18T04:27:23.0692810Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:27:23.0702334Z 
2022-05-18T04:27:23.0702485Z Running tests...
2022-05-18T04:27:23.0703251Z ----------------------------------------------------------------------
2022-05-18T04:27:23.3511853Z   test_sparse_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21549
2022-05-18T04:27:23.3533552Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21550
2022-05-18T04:27:23.3556579Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21551
2022-05-18T04:27:24.1669068Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:27:24.1669484Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:27:24.1669862Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:27:24.1670473Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:27:24.1671190Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:27:24.1671787Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:27:24.1774941Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:27:24.2682085Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:27:24.2682696Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:27:24.4608275Z skip: CUDA is not available. (1.390s)
2022-05-18T04:27:24.4608532Z 
2022-05-18T04:27:24.4608835Z ----------------------------------------------------------------------
2022-05-18T04:27:24.4609088Z Ran 1 test in 1.390s
2022-05-18T04:27:24.4609205Z 
2022-05-18T04:27:24.4609278Z OK (skipped=1)
2022-05-18T04:27:24.4609409Z 
2022-05-18T04:27:24.4609495Z Generating XML reports...
2022-05-18T04:27:24.4641848Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042723.xml
2022-05-18T04:27:25.3839985Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:27:25.3850417Z 
2022-05-18T04:27:25.3850900Z Running tests...
2022-05-18T04:27:25.3851320Z ----------------------------------------------------------------------
2022-05-18T04:27:25.6665838Z   test_stateless_api_with_ddp (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21602
2022-05-18T04:27:25.6688023Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21603
2022-05-18T04:27:25.6711477Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21604
2022-05-18T04:27:26.4855298Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:27:26.4855825Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:27:26.4856212Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:27:26.4856857Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:27:26.4857447Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:27:26.4857966Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:27:26.4964166Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:27:26.4964564Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:27:26.5868366Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:27:26.7761537Z skip: Need at least 2 CUDA devices (1.391s)
2022-05-18T04:27:26.7761823Z 
2022-05-18T04:27:26.7762223Z ----------------------------------------------------------------------
2022-05-18T04:27:26.7762492Z Ran 1 test in 1.391s
2022-05-18T04:27:26.7762605Z 
2022-05-18T04:27:26.7762665Z OK (skipped=1)
2022-05-18T04:27:26.7762775Z 
2022-05-18T04:27:26.7762861Z Generating XML reports...
2022-05-18T04:27:26.7795457Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042725.xml
2022-05-18T04:27:27.7010347Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:27:27.7020541Z 
2022-05-18T04:27:27.7020674Z Running tests...
2022-05-18T04:27:27.7021066Z ----------------------------------------------------------------------
2022-05-18T04:27:27.9849732Z   test_static_graph_api_cpu (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21655
2022-05-18T04:27:27.9872642Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21656
2022-05-18T04:27:27.9895998Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21657
2022-05-18T04:27:28.8157669Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:27:28.8257817Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:27:28.8258378Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:27:28.8259103Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:27:28.8259634Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:27:28.8260169Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:27:28.8367074Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:27:28.8368012Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:27:28.8430147Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpblub8hfd
2022-05-18T04:27:28.8431938Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpblub8hfd/_remote_module_non_scriptable.py
2022-05-18T04:27:28.8432601Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2ky8zxh2
2022-05-18T04:27:28.8433860Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2ky8zxh2/_remote_module_non_scriptable.py
2022-05-18T04:27:28.9271181Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:27:28.9335320Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8my6wc7q
2022-05-18T04:27:28.9337477Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8my6wc7q/_remote_module_non_scriptable.py
2022-05-18T04:27:29.0945516Z ok (1.392s)
2022-05-18T04:27:29.0945753Z 
2022-05-18T04:27:29.0946209Z ----------------------------------------------------------------------
2022-05-18T04:27:29.0946610Z Ran 1 test in 1.392s
2022-05-18T04:27:29.0946781Z 
2022-05-18T04:27:29.0946873Z OK
2022-05-18T04:27:29.0947026Z 
2022-05-18T04:27:29.0947177Z Generating XML reports...
2022-05-18T04:27:29.0978555Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042727.xml
2022-05-18T04:27:30.0159008Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:27:30.0169691Z 
2022-05-18T04:27:30.0170173Z Running tests...
2022-05-18T04:27:30.0170588Z ----------------------------------------------------------------------
2022-05-18T04:27:30.2978696Z   test_sync_bn_logged (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21708
2022-05-18T04:27:30.3001251Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21709
2022-05-18T04:27:30.3024494Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21710
2022-05-18T04:27:31.1185723Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:27:31.1186447Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:27:31.1187134Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:27:31.1187981Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:27:31.1188511Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:27:31.1189274Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:27:31.1195805Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:27:31.1196353Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:27:31.1197205Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:27:31.3074253Z skip: Need at least 2 CUDA devices (1.290s)
2022-05-18T04:27:31.3074562Z 
2022-05-18T04:27:31.3075059Z ----------------------------------------------------------------------
2022-05-18T04:27:31.3075331Z Ran 1 test in 1.290s
2022-05-18T04:27:31.3075445Z 
2022-05-18T04:27:31.3075521Z OK (skipped=1)
2022-05-18T04:27:31.3075690Z 
2022-05-18T04:27:31.3075764Z Generating XML reports...
2022-05-18T04:27:31.3107856Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042730.xml
2022-05-18T04:27:32.2256903Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:27:32.2267026Z 
2022-05-18T04:27:32.2267159Z Running tests...
2022-05-18T04:27:32.2267741Z ----------------------------------------------------------------------
2022-05-18T04:27:32.5117598Z   test_undefined_grad_parity_unused_parameters (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21761
2022-05-18T04:27:32.5139977Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21762
2022-05-18T04:27:32.5162848Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21763
2022-05-18T04:27:33.3380754Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:27:33.3381430Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:27:33.3381819Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:27:33.3382456Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:27:33.3383155Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:27:33.3383684Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:27:33.3392317Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:27:33.3392892Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:27:33.3393283Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:27:33.5212214Z skip: Need at least 2 CUDA devices (1.294s)
2022-05-18T04:27:33.5212637Z 
2022-05-18T04:27:33.5213387Z ----------------------------------------------------------------------
2022-05-18T04:27:33.5213688Z Ran 1 test in 1.294s
2022-05-18T04:27:33.5213808Z 
2022-05-18T04:27:33.5213873Z OK (skipped=1)
2022-05-18T04:27:33.5213988Z 
2022-05-18T04:27:33.5214074Z Generating XML reports...
2022-05-18T04:27:33.5245075Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042732.xml
2022-05-18T04:27:34.4570915Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:27:34.4580662Z 
2022-05-18T04:27:34.4580795Z Running tests...
2022-05-18T04:27:34.4581359Z ----------------------------------------------------------------------
2022-05-18T04:27:34.4596392Z   test_verify_model_across_rank_with_logger (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'nccl'} (0.001s)
2022-05-18T04:27:34.4597018Z 
2022-05-18T04:27:34.4597233Z ----------------------------------------------------------------------
2022-05-18T04:27:34.4597480Z Ran 1 test in 0.002s
2022-05-18T04:27:34.4597688Z 
2022-05-18T04:27:34.4597773Z OK (skipped=1)
2022-05-18T04:27:34.4597882Z 
2022-05-18T04:27:34.4597968Z Generating XML reports...
2022-05-18T04:27:34.4628880Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042734.xml
2022-05-18T04:27:35.3019005Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:27:35.3029868Z 
2022-05-18T04:27:35.3030169Z Running tests...
2022-05-18T04:27:35.3030803Z ----------------------------------------------------------------------
2022-05-18T04:27:35.3046122Z   test_verify_model_across_rank_without_logger (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'nccl'} (0.001s)
2022-05-18T04:27:35.3046441Z 
2022-05-18T04:27:35.3046662Z ----------------------------------------------------------------------
2022-05-18T04:27:35.3046986Z Ran 1 test in 0.002s
2022-05-18T04:27:35.3047089Z 
2022-05-18T04:27:35.3047170Z OK (skipped=1)
2022-05-18T04:27:35.3047281Z 
2022-05-18T04:27:35.3047369Z Generating XML reports...
2022-05-18T04:27:35.3078263Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042735.xml
2022-05-18T04:27:35.5252799Z Running distributed tests for the gloo backend with file init_method
2022-05-18T04:27:35.5254647Z Executing ['/opt/conda/bin/python', 'distributed/test_distributed_spawn.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-05-18 04:27:35.525217]
2022-05-18T04:27:36.2616788Z <unittest.suite.TestSuite tests=[]>
2022-05-18T04:27:36.2656335Z <unittest.suite.TestSuite tests=[<__main__.TestDistBackendWithSpawn testMethod=test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager>, <__main__.TestDistBackendWithSpawn testMethod=test_3_level_hierarchical_model_averager>, <__main__.TestDistBackendWithSpawn testMethod=test_Backend_enum_class>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallelCPU>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallelCPU_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_2D_Input>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Channels_Last>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_No_Affine>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_non_default_stream>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_requires_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_with_amp_and_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedSampler_padding>, <__main__.TestDistBackendWithSpawn testMethod=test_SyncBatchNorm_process_group>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_allreduce_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_allreduce_with_then_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_simple>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_with_empty>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_multigpu_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_object_default_pg>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_object_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_max_complex_unsupported>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_complex_unsupported_ops>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_multigpu_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_result_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_async>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda_async>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_average_parameters>, <__main__.TestDistBackendWithSpawn testMethod=test_backend_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_backend_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_global>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_group>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_gloo>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_gloo_tags>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_mixed_backend_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_no_rank_zero_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_op_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_op_list_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_ring_exchange_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_self_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_tensor_err>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_group>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_object_list>, <__main__.TestDistBackendWithSpawn testMethod=test_compute_bucket_assignment_by_size_sparse_error_with_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_compute_bucket_assignment_by_size_sparse_error_without_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_broadcast_buffer>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_broadcast_buffer_via_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_buffer_hook_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_buffer_hook_allreduce_return_future>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_build_debug_param_to_name_mapping>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_build_debug_param_to_name_mapping_requires_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_comm_hook_logging>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_control_flow_different_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_control_flow_same_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_create_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_device>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_forward_backward_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_grad_div_uneven_inputs>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_allreduce_process_group>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_post_localSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_powerSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_ignore_params_arg>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_inference>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_join_model_equivalence>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_logging_data_cpu>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_logging_data_gpu>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_model_diff_num_params_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_model_diff_shape_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_multiple_nested_unused_params_err_ignore_params>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_multiple_nested_unused_params_error>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_namedtuple>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_new_tensor_in_fwd>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_new_tensor_in_fwd_static_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_profiling_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_profiling_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_python_error_logged>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_returns_tensor_with_no_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_shared_grad_acc_unused_params>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_static_graph_nested_types>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_sync_bn_training_vs_eval>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_sync_module_states>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_input_exception>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_input_join_disable>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_inputs>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_inputs_stop_iteration_sync_bn>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_unused_params_rebuild_buckets_exception>, <__main__.TestDistBackendWithSpawn testMethod=test_destroy_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_destroy_group>, <__main__.TestDistBackendWithSpawn testMethod=test_detect_ddp_is_actually_static>, <__main__.TestDistBackendWithSpawn testMethod=test_different_graph_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_dump_DDP_relevant_env_vars>, <__main__.TestDistBackendWithSpawn testMethod=test_gather>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_checks>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_group>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_object>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_object_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_get_backend>, <__main__.TestDistBackendWithSpawn testMethod=test_get_future>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank_size_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank_size_group>, <__main__.TestDistBackendWithSpawn testMethod=test_invalid_static_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_irecv>, <__main__.TestDistBackendWithSpawn testMethod=test_isend>, <__main__.TestDistBackendWithSpawn testMethod=test_isend_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_isend_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_allreduce_hang>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_allreduce_hang_wait_all_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_failure_order>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo_rank_0_timeout>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_wait_all_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_allgather>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_broadcast>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_reduce>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_high_priority_stream>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration_input_rank_exceeds_world_size>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration_negative_input_rank>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_group_size_exceeds_world_size>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_overlap_not_allowed>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_world_size_not_divisible_by_group_size>, <__main__.TestDistBackendWithSpawn testMethod=test_output_unused_in_loss_dict_module>, <__main__.TestDistBackendWithSpawn testMethod=test_output_unused_in_loss_tuple_module>, <__main__.TestDistBackendWithSpawn testMethod=test_periodic_model_averager>, <__main__.TestDistBackendWithSpawn testMethod=test_periodic_model_averager_param_group>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity_with_hierarchical_sgd>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_cuda_twice>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_twice>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_checks>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_group>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_object_list>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_sparse_all_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_sparse_all_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_stateless_api_with_ddp>, <__main__.TestDistBackendWithSpawn testMethod=test_static_graph_api_cpu>, <__main__.TestDistBackendWithSpawn testMethod=test_sync_bn_logged>, <__main__.TestDistBackendWithSpawn testMethod=test_undefined_grad_parity_unused_parameters>, <__main__.TestDistBackendWithSpawn testMethod=test_verify_model_across_rank_with_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_verify_model_across_rank_without_logger>]>
2022-05-18T04:27:36.2682988Z test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2683504Z test_3_level_hierarchical_model_averager (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2683818Z test_Backend_enum_class (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2684124Z test_DistributedDataParallel (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2684439Z test_DistributedDataParallelCPU (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2684788Z test_DistributedDataParallelCPU_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2685146Z test_DistributedDataParallel_SyncBatchNorm (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2685498Z test_DistributedDataParallel_SyncBatchNorm_2D_Input (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2685879Z test_DistributedDataParallel_SyncBatchNorm_Channels_Last (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2686283Z test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2686696Z test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2687072Z test_DistributedDataParallel_SyncBatchNorm_No_Affine (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2687463Z test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2687845Z test_DistributedDataParallel_non_default_stream (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2688197Z test_DistributedDataParallel_requires_grad (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2688540Z test_DistributedDataParallel_with_amp_and_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2688879Z test_DistributedSampler_padding (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2689196Z test_SyncBatchNorm_process_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2689496Z test_accumulate_gradients_no_sync (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2689823Z test_accumulate_gradients_no_sync_allreduce_hook (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2690177Z test_accumulate_gradients_no_sync_allreduce_with_then_hook (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2690522Z test_accumulate_gradients_no_sync_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2690806Z test_all_gather (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2691093Z test_all_gather_coalesced_complex (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2691405Z test_all_gather_coalesced_full_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2691700Z test_all_gather_coalesced_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2692006Z test_all_gather_coalesced_simple (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2692318Z test_all_gather_coalesced_with_empty (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2692608Z test_all_gather_complex (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2692889Z test_all_gather_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2693174Z test_all_gather_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2693464Z test_all_gather_full_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2693735Z test_all_gather_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2694022Z test_all_gather_multigpu (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2694320Z test_all_gather_multigpu_complex (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2694612Z test_all_gather_object_default_pg (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2694911Z test_all_gather_object_subgroup (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2695280Z test_all_reduce_coalesced_full_group_max (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2695601Z test_all_reduce_coalesced_full_group_min (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2695944Z test_all_reduce_coalesced_full_group_product (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2696273Z test_all_reduce_coalesced_full_group_sum (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2696588Z test_all_reduce_coalesced_group_max (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2696887Z test_all_reduce_coalesced_group_min (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2697202Z test_all_reduce_coalesced_group_product (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2697518Z test_all_reduce_coalesced_group_sum (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2697809Z test_all_reduce_coalesced_max (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2698135Z test_all_reduce_coalesced_max_complex_unsupported (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2698456Z test_all_reduce_coalesced_min (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2698755Z test_all_reduce_coalesced_product (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2699044Z test_all_reduce_coalesced_sum (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2699353Z test_all_reduce_complex_unsupported_ops (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2699664Z test_all_reduce_full_group_max (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2699947Z test_all_reduce_full_group_min (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2700250Z test_all_reduce_full_group_product (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2700549Z test_all_reduce_full_group_sum (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2700838Z test_all_reduce_group_max (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2701109Z test_all_reduce_group_min (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2701400Z test_all_reduce_group_product (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2701691Z test_all_reduce_group_sum (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2701955Z test_all_reduce_max (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2702227Z test_all_reduce_min (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2702507Z test_all_reduce_multigpu (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2702790Z test_all_reduce_multigpu_complex (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2703191Z test_all_reduce_product (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2703480Z test_all_reduce_result_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2703758Z test_all_reduce_sum (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2704024Z test_all_reduce_sum_async (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2704314Z test_all_reduce_sum_complex (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2704599Z test_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2704878Z test_all_reduce_sum_cuda_async (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2705179Z test_all_reduce_sum_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2705466Z test_all_to_all (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2705729Z test_all_to_all_complex (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2706010Z test_all_to_all_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2706299Z test_all_to_all_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2706586Z test_all_to_all_full_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2706870Z test_all_to_all_full_group_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2707154Z test_all_to_all_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2707431Z test_all_to_all_group_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2707714Z test_all_to_all_single_equal_split (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2708091Z test_all_to_all_single_equal_split_complex (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2708409Z test_all_to_all_single_equal_split_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2708763Z test_all_to_all_single_equal_split_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2709099Z test_all_to_all_single_equal_split_full_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2709433Z test_all_to_all_single_equal_split_full_group_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2709762Z test_all_to_all_single_equal_split_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2710072Z test_all_to_all_single_equal_split_group_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2710388Z test_all_to_all_single_unequal_split (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2710704Z test_all_to_all_single_unequal_split_complex (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2711011Z test_all_to_all_single_unequal_split_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2711340Z test_all_to_all_single_unequal_split_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2711680Z test_all_to_all_single_unequal_split_full_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2712017Z test_all_to_all_single_unequal_split_full_group_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2712334Z test_all_to_all_single_unequal_split_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2712660Z test_all_to_all_single_unequal_split_group_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2712965Z test_average_parameters (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2713241Z test_backend_full_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2713518Z test_backend_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2713782Z test_barrier (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2714046Z test_barrier_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2714314Z test_barrier_full_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2714598Z test_barrier_full_group_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2714882Z test_barrier_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2715148Z test_barrier_group_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2715437Z test_barrier_timeout_full_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2715794Z test_barrier_timeout_global (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2716074Z test_barrier_timeout_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2716366Z test_batch_isend_irecv_gloo (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2716664Z test_batch_isend_irecv_gloo_tags (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2716977Z test_batch_isend_irecv_mixed_backend_err (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2717268Z test_batch_isend_irecv_nccl (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2717573Z test_batch_isend_irecv_no_rank_zero_nccl (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2717877Z test_batch_isend_irecv_op_err (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2718166Z test_batch_isend_irecv_op_list_err (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2718482Z test_batch_isend_irecv_ring_exchange_nccl (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2718793Z test_batch_isend_irecv_self_nccl (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2719082Z test_batch_isend_irecv_tensor_err (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2719367Z test_broadcast (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2719638Z test_broadcast_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2719922Z test_broadcast_full_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2720192Z test_broadcast_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2720474Z test_broadcast_multigpu (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2720804Z test_broadcast_object_list (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2721157Z test_compute_bucket_assignment_by_size_sparse_error_with_logger (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2721535Z test_compute_bucket_assignment_by_size_sparse_error_without_logger (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2721864Z test_ddp_broadcast_buffer (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2722167Z test_ddp_broadcast_buffer_via_hook (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2722460Z test_ddp_buffer_hook_allreduce (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2722779Z test_ddp_buffer_hook_allreduce_return_future (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2723109Z test_ddp_build_debug_param_to_name_mapping (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2723438Z test_ddp_build_debug_param_to_name_mapping_requires_grad (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2723764Z test_ddp_comm_hook_logging (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2724073Z test_ddp_control_flow_different_across_ranks (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2724385Z test_ddp_control_flow_same_across_ranks (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2724682Z test_ddp_create_graph (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2724951Z test_ddp_device (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2725234Z test_ddp_forward_backward_hook (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2725521Z test_ddp_grad_div_uneven_inputs (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2725824Z test_ddp_hook_parity_allreduce (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2726143Z test_ddp_hook_parity_allreduce_process_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2726453Z test_ddp_hook_parity_post_localSGD (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2726758Z test_ddp_hook_parity_powerSGD (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2727096Z test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2727465Z test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2727871Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2728325Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2728772Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2729225Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2729656Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2730105Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2730546Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2730987Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2731378Z test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2731742Z test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2732065Z test_ddp_ignore_params_arg (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2732344Z test_ddp_inference (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2732657Z test_ddp_join_model_equivalence (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2732985Z test_ddp_logging_data_cpu (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2733275Z test_ddp_logging_data_gpu (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2733573Z test_ddp_model_diff_num_params_across_ranks (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2733893Z test_ddp_model_diff_shape_across_ranks (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2734231Z test_ddp_multiple_nested_unused_params_err_ignore_params (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2734566Z test_ddp_multiple_nested_unused_params_error (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2734870Z test_ddp_namedtuple (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2735156Z test_ddp_new_tensor_in_fwd (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2735460Z test_ddp_new_tensor_in_fwd_static_graph (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2735769Z test_ddp_profiling_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2736084Z test_ddp_profiling_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2736389Z test_ddp_python_error_logged (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2736679Z test_ddp_returns_tensor_with_no_grad (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2736990Z test_ddp_shared_grad_acc_unused_params (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2737303Z test_ddp_static_graph_nested_types (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2737607Z test_ddp_sync_bn_training_vs_eval (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2737893Z test_ddp_sync_module_states (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2738189Z test_ddp_uneven_input_exception (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2738493Z test_ddp_uneven_input_join_disable (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2738779Z test_ddp_uneven_inputs (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2739083Z test_ddp_uneven_inputs_stop_iteration_sync_bn (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2739419Z test_ddp_unused_params_rebuild_buckets_exception (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2739719Z test_destroy_full_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2739995Z test_destroy_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2740284Z test_detect_ddp_is_actually_static (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2740589Z test_different_graph_across_ranks (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2740878Z test_dump_DDP_relevant_env_vars (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2741159Z test_gather (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2741423Z test_gather_checks (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2741678Z test_gather_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2741956Z test_gather_full_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2742229Z test_gather_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2742489Z test_gather_object (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2742772Z test_gather_object_subgroup (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2743201Z test_get_backend (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2743468Z test_get_future (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2743717Z test_get_rank (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2743995Z test_get_rank_size_full_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2744284Z test_get_rank_size_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2744556Z test_invalid_static_graph (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2744825Z test_irecv (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2745075Z test_isend (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2745415Z test_isend_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2745711Z test_isend_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2746055Z test_monitored_barrier_allreduce_hang (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2746397Z test_monitored_barrier_allreduce_hang_wait_all_ranks (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2746717Z test_monitored_barrier_failure_order (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2747023Z test_monitored_barrier_gloo (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2747328Z test_monitored_barrier_gloo_rank_0_timeout (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2747633Z test_monitored_barrier_gloo_subgroup (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2747946Z test_monitored_barrier_wait_all_ranks (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2748261Z test_nccl_backend_bool_allgather (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2748559Z test_nccl_backend_bool_allreduce (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2748865Z test_nccl_backend_bool_broadcast (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2749165Z test_nccl_backend_bool_reduce (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2749464Z test_nccl_high_priority_stream (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2749741Z test_new_subgroups (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2750030Z test_new_subgroups_by_enumeration (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2750372Z test_new_subgroups_by_enumeration_input_rank_exceeds_world_size (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2750718Z test_new_subgroups_by_enumeration_negative_input_rank (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2751064Z test_new_subgroups_group_size_exceeds_world_size (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2751388Z test_new_subgroups_overlap_not_allowed (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2751726Z test_new_subgroups_world_size_not_divisible_by_group_size (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2752046Z test_output_unused_in_loss_dict_module (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2752361Z test_output_unused_in_loss_tuple_module (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2752668Z test_periodic_model_averager (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2752967Z test_periodic_model_averager_param_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2753286Z test_post_localSGD_optimizer_parity (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2753613Z test_post_localSGD_optimizer_parity_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2753965Z test_post_localSGD_optimizer_parity_with_hierarchical_sgd (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2754327Z test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2754670Z test_reduce_full_group_max (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2754956Z test_reduce_full_group_min (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2755242Z test_reduce_full_group_product (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2755594Z test_reduce_full_group_sum (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2755884Z test_reduce_group_max (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2756162Z test_reduce_group_min (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2756434Z test_reduce_group_product (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2756716Z test_reduce_group_sum (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2756987Z test_reduce_max (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2757237Z test_reduce_min (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2757510Z test_reduce_multigpu (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2757788Z test_reduce_product (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2758086Z test_reduce_sum (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2758358Z test_reduce_sum_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2758695Z test_reduce_sum_cuda_twice (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2758966Z test_reduce_sum_twice (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2759232Z test_scatter (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2759497Z test_scatter_checks (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2759771Z test_scatter_complex (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2760027Z test_scatter_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2760304Z test_scatter_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2760591Z test_scatter_full_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2760857Z test_scatter_group (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2761140Z test_scatter_object_list (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2761414Z test_send_recv (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2761673Z test_send_recv_any_source (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2761982Z test_send_recv_any_source_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2762307Z test_send_recv_any_source_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2762616Z test_send_recv_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2762891Z test_send_recv_nccl (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2763183Z test_send_recv_nccl_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2763493Z test_send_recv_nccl_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2763783Z test_send_recv_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2764071Z test_send_recv_with_tag (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2764369Z test_send_recv_with_tag_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2764677Z test_send_recv_with_tag_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2764982Z test_sparse_all_reduce_sum (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2765275Z test_sparse_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2765571Z test_stateless_api_with_ddp (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2765848Z test_static_graph_api_cpu (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2766126Z test_sync_bn_logged (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2766432Z test_undefined_grad_parity_unused_parameters (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2766748Z test_verify_model_across_rank_with_logger (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:36.2767067Z test_verify_model_across_rank_without_logger (__main__.TestDistBackendWithSpawn)
2022-05-18T04:27:37.0008049Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:27:37.0017709Z 
2022-05-18T04:27:37.0017823Z Running tests...
2022-05-18T04:27:37.0018778Z ----------------------------------------------------------------------
2022-05-18T04:27:37.2878749Z   test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21844
2022-05-18T04:27:37.2901663Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21845
2022-05-18T04:27:37.2925095Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21846
2022-05-18T04:27:38.1231474Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:27:38.1232047Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:27:38.1232437Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:27:38.1233236Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:27:38.1233826Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:27:38.1234353Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:27:38.1338711Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:27:38.2245723Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:27:38.2246293Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:27:38.3975156Z skip: Need at least 2 CUDA devices (1.395s)
2022-05-18T04:27:38.3975466Z 
2022-05-18T04:27:38.3975778Z ----------------------------------------------------------------------
2022-05-18T04:27:38.3976068Z Ran 1 test in 1.396s
2022-05-18T04:27:38.3976169Z 
2022-05-18T04:27:38.3976243Z OK (skipped=1)
2022-05-18T04:27:38.3976350Z 
2022-05-18T04:27:38.3976439Z Generating XML reports...
2022-05-18T04:27:38.4007321Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042736.xml
2022-05-18T04:27:39.3311711Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:27:39.3321904Z 
2022-05-18T04:27:39.3322229Z Running tests...
2022-05-18T04:27:39.3322849Z ----------------------------------------------------------------------
2022-05-18T04:27:39.3356459Z   test_3_level_hierarchical_model_averager (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.003s)
2022-05-18T04:27:39.3356815Z 
2022-05-18T04:27:39.3357092Z ----------------------------------------------------------------------
2022-05-18T04:27:39.3357345Z Ran 1 test in 0.003s
2022-05-18T04:27:39.3357481Z 
2022-05-18T04:27:39.3357556Z OK (skipped=1)
2022-05-18T04:27:39.3357667Z 
2022-05-18T04:27:39.3357739Z Generating XML reports...
2022-05-18T04:27:39.3388760Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042739.xml
2022-05-18T04:27:40.1689090Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:27:40.1698316Z 
2022-05-18T04:27:40.1698412Z Running tests...
2022-05-18T04:27:40.1698807Z ----------------------------------------------------------------------
2022-05-18T04:27:40.4513132Z   test_Backend_enum_class (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21907
2022-05-18T04:27:40.4536119Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21908
2022-05-18T04:27:40.4560112Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21909
2022-05-18T04:27:41.2961425Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:27:41.3004868Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:27:41.3005514Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:27:41.3006155Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:27:41.3006682Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:27:41.3062573Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:27:41.3114760Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:27:41.3115439Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:27:41.4073623Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:27:41.6612798Z ok (1.491s)
2022-05-18T04:27:41.6613387Z 
2022-05-18T04:27:41.6613749Z ----------------------------------------------------------------------
2022-05-18T04:27:41.6614056Z Ran 1 test in 1.491s
2022-05-18T04:27:41.6614263Z 
2022-05-18T04:27:41.6614349Z OK
2022-05-18T04:27:41.6614639Z 
2022-05-18T04:27:41.6614746Z Generating XML reports...
2022-05-18T04:27:41.6646816Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042740.xml
2022-05-18T04:27:42.5858363Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:27:42.5869539Z 
2022-05-18T04:27:42.5869718Z Running tests...
2022-05-18T04:27:42.5870379Z ----------------------------------------------------------------------
2022-05-18T04:27:42.8604186Z   test_DistributedDataParallel (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77317 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure IN_CI is not set and you are not using the flag --import-disabled-tests. (0.273s)
2022-05-18T04:27:42.8604703Z 
2022-05-18T04:27:42.8604912Z ----------------------------------------------------------------------
2022-05-18T04:27:42.8605180Z Ran 1 test in 0.273s
2022-05-18T04:27:42.8605292Z 
2022-05-18T04:27:42.8605367Z OK (skipped=1)
2022-05-18T04:27:42.8605474Z 
2022-05-18T04:27:42.8605547Z Generating XML reports...
2022-05-18T04:27:42.8632077Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042742.xml
2022-05-18T04:27:43.7584262Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:27:43.7594353Z 
2022-05-18T04:27:43.7594508Z Running tests...
2022-05-18T04:27:43.7595566Z ----------------------------------------------------------------------
2022-05-18T04:27:44.0393158Z   test_DistributedDataParallelCPU (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21970
2022-05-18T04:27:44.0413613Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21971
2022-05-18T04:27:44.0436226Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21972
2022-05-18T04:27:44.8721915Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:27:44.8804948Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:27:44.8805519Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:27:44.8806580Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:27:44.8807171Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:27:44.8822684Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:27:44.8914780Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:27:44.8915632Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:27:44.8984023Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0freivw9
2022-05-18T04:27:44.8984630Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi6evh8ai
2022-05-18T04:27:44.8985249Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0freivw9/_remote_module_non_scriptable.py
2022-05-18T04:27:44.8985931Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi6evh8ai/_remote_module_non_scriptable.py
2022-05-18T04:27:44.9835705Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:27:44.9904845Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi__byrz0
2022-05-18T04:27:44.9906394Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi__byrz0/_remote_module_non_scriptable.py
2022-05-18T04:27:45.0060919Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:27:45.0061313Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:27:45.0061676Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:27:45.3491912Z ok (1.589s)
2022-05-18T04:27:45.3492142Z 
2022-05-18T04:27:45.3492601Z ----------------------------------------------------------------------
2022-05-18T04:27:45.3492984Z Ran 1 test in 1.590s
2022-05-18T04:27:45.3493188Z 
2022-05-18T04:27:45.3493286Z OK
2022-05-18T04:27:45.3493428Z 
2022-05-18T04:27:45.3493577Z Generating XML reports...
2022-05-18T04:27:45.3525326Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042743.xml
2022-05-18T04:27:46.2912418Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:27:46.2922720Z 
2022-05-18T04:27:46.2922817Z Running tests...
2022-05-18T04:27:46.2923247Z ----------------------------------------------------------------------
2022-05-18T04:27:46.5736371Z   test_DistributedDataParallelCPU_grad_is_view (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22023
2022-05-18T04:27:46.5758306Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22024
2022-05-18T04:27:46.5780758Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22025
2022-05-18T04:27:47.4266330Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:27:47.4366993Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:27:47.4367395Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:27:47.4368026Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:27:47.4368553Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:27:47.4369073Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:27:47.4477238Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:27:47.4477845Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:27:47.4547857Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfsldo60a
2022-05-18T04:27:47.4548543Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7ojie2qg
2022-05-18T04:27:47.4549749Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfsldo60a/_remote_module_non_scriptable.py
2022-05-18T04:27:47.4550418Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7ojie2qg/_remote_module_non_scriptable.py
2022-05-18T04:27:47.5379492Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:27:47.5446871Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf7alcz3x
2022-05-18T04:27:47.5448893Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf7alcz3x/_remote_module_non_scriptable.py
2022-05-18T04:27:47.5602091Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:27:47.5602689Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:27:47.5603114Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:27:47.8836326Z ok (1.591s)
2022-05-18T04:27:47.8836556Z 
2022-05-18T04:27:47.8837073Z ----------------------------------------------------------------------
2022-05-18T04:27:47.8837328Z Ran 1 test in 1.591s
2022-05-18T04:27:47.8837455Z 
2022-05-18T04:27:47.8837515Z OK
2022-05-18T04:27:47.8837608Z 
2022-05-18T04:27:47.8837687Z Generating XML reports...
2022-05-18T04:27:47.8867806Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042746.xml
2022-05-18T04:27:48.7987111Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:27:48.7997192Z 
2022-05-18T04:27:48.7997656Z Running tests...
2022-05-18T04:27:48.7998067Z ----------------------------------------------------------------------
2022-05-18T04:27:49.0794738Z   test_DistributedDataParallel_SyncBatchNorm (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22076
2022-05-18T04:27:49.0816747Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22077
2022-05-18T04:27:49.0840297Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22078
2022-05-18T04:27:49.9342393Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:27:49.9438261Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:27:49.9438689Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:27:49.9439299Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:27:49.9439844Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:27:49.9443441Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:27:49.9546783Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:27:49.9547468Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:27:50.0456505Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:27:50.1890259Z skip: CUDA is not available. (1.389s)
2022-05-18T04:27:50.1890576Z 
2022-05-18T04:27:50.1891068Z ----------------------------------------------------------------------
2022-05-18T04:27:50.1891414Z Ran 1 test in 1.389s
2022-05-18T04:27:50.1891530Z 
2022-05-18T04:27:50.1891603Z OK (skipped=1)
2022-05-18T04:27:50.1891709Z 
2022-05-18T04:27:50.1891782Z Generating XML reports...
2022-05-18T04:27:50.1922634Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042748.xml
2022-05-18T04:27:51.1028237Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:27:51.1038539Z 
2022-05-18T04:27:51.1038853Z Running tests...
2022-05-18T04:27:51.1039530Z ----------------------------------------------------------------------
2022-05-18T04:27:51.3852529Z   test_DistributedDataParallel_SyncBatchNorm_2D_Input (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22129
2022-05-18T04:27:51.3874911Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22130
2022-05-18T04:27:51.3897454Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22131
2022-05-18T04:27:52.2101961Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:27:52.2203040Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:27:52.2203762Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:27:52.2204408Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:27:52.2204925Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:27:52.2205452Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:27:52.2313478Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:27:52.2314057Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:27:52.3217391Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:27:52.4948338Z skip: CUDA is not available. (1.391s)
2022-05-18T04:27:52.4948519Z 
2022-05-18T04:27:52.4948856Z ----------------------------------------------------------------------
2022-05-18T04:27:52.4949175Z Ran 1 test in 1.391s
2022-05-18T04:27:52.4949276Z 
2022-05-18T04:27:52.4949348Z OK (skipped=1)
2022-05-18T04:27:52.4949456Z 
2022-05-18T04:27:52.4949542Z Generating XML reports...
2022-05-18T04:27:52.4980514Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042751.xml
2022-05-18T04:27:53.4192370Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:27:53.4202415Z 
2022-05-18T04:27:53.4202666Z Running tests...
2022-05-18T04:27:53.4203001Z ----------------------------------------------------------------------
2022-05-18T04:27:53.7018680Z   test_DistributedDataParallel_SyncBatchNorm_Channels_Last (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22182
2022-05-18T04:27:53.7040169Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22183
2022-05-18T04:27:53.7062356Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22184
2022-05-18T04:27:54.5181930Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:27:54.5283213Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:27:54.5283866Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:27:54.5284580Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:27:54.5285117Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:27:54.5285663Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:27:54.5294037Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:27:54.5296088Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:27:54.5296798Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:27:54.7112260Z skip: CUDA is not available. (1.291s)
2022-05-18T04:27:54.7112493Z 
2022-05-18T04:27:54.7113002Z ----------------------------------------------------------------------
2022-05-18T04:27:54.7113395Z Ran 1 test in 1.291s
2022-05-18T04:27:54.7113510Z 
2022-05-18T04:27:54.7113584Z OK (skipped=1)
2022-05-18T04:27:54.7113692Z 
2022-05-18T04:27:54.7113776Z Generating XML reports...
2022-05-18T04:27:54.7144219Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042753.xml
2022-05-18T04:27:55.6446263Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:27:55.6455506Z 
2022-05-18T04:27:55.6455645Z Running tests...
2022-05-18T04:27:55.6456595Z ----------------------------------------------------------------------
2022-05-18T04:27:55.9252911Z   test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22235
2022-05-18T04:27:55.9274950Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22236
2022-05-18T04:27:55.9297386Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22237
2022-05-18T04:27:56.7080535Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:27:56.7182260Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:27:56.7183379Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:27:56.7183811Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:27:56.7184315Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:27:56.7184834Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:27:56.7195979Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:27:56.7196769Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:27:56.7197440Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:27:56.9345025Z skip: CUDA is not available. (1.289s)
2022-05-18T04:27:56.9345315Z 
2022-05-18T04:27:56.9345835Z ----------------------------------------------------------------------
2022-05-18T04:27:56.9346141Z Ran 1 test in 1.289s
2022-05-18T04:27:56.9346258Z 
2022-05-18T04:27:56.9346451Z OK (skipped=1)
2022-05-18T04:27:56.9346623Z 
2022-05-18T04:27:56.9346717Z Generating XML reports...
2022-05-18T04:27:56.9376788Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042755.xml
2022-05-18T04:27:57.8520786Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:27:57.8530263Z 
2022-05-18T04:27:57.8530370Z Running tests...
2022-05-18T04:27:57.8530815Z ----------------------------------------------------------------------
2022-05-18T04:27:58.1324604Z   test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22288
2022-05-18T04:27:58.1346569Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22289
2022-05-18T04:27:58.1369477Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22290
2022-05-18T04:27:58.9271931Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:27:58.9272358Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:27:58.9272737Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:27:58.9273336Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:27:58.9273875Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:27:58.9274407Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:27:58.9378598Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:27:59.0285928Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:27:59.0286582Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:27:59.2419564Z skip: CUDA is not available. (1.389s)
2022-05-18T04:27:59.2419860Z 
2022-05-18T04:27:59.2420369Z ----------------------------------------------------------------------
2022-05-18T04:27:59.2420818Z Ran 1 test in 1.389s
2022-05-18T04:27:59.2421025Z 
2022-05-18T04:27:59.2421162Z OK (skipped=1)
2022-05-18T04:27:59.2421353Z 
2022-05-18T04:27:59.2421489Z Generating XML reports...
2022-05-18T04:27:59.2451669Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042757.xml
2022-05-18T04:28:00.1624854Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:28:00.1634305Z 
2022-05-18T04:28:00.1634412Z Running tests...
2022-05-18T04:28:00.1634991Z ----------------------------------------------------------------------
2022-05-18T04:28:00.4442608Z   test_DistributedDataParallel_SyncBatchNorm_No_Affine (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22341
2022-05-18T04:28:00.4465087Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22342
2022-05-18T04:28:00.4487782Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22343
2022-05-18T04:28:01.2932229Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:28:01.2932651Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:28:01.2933433Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:28:01.2934128Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:28:01.2934956Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:28:01.2935804Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:28:01.3038884Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:28:01.3946454Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:28:01.3947038Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:28:01.5539691Z skip: CUDA is not available. (1.390s)
2022-05-18T04:28:01.5539992Z 
2022-05-18T04:28:01.5540496Z ----------------------------------------------------------------------
2022-05-18T04:28:01.5540792Z Ran 1 test in 1.390s
2022-05-18T04:28:01.5540909Z 
2022-05-18T04:28:01.5540968Z OK (skipped=1)
2022-05-18T04:28:01.5541079Z 
2022-05-18T04:28:01.5541165Z Generating XML reports...
2022-05-18T04:28:01.5571339Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042800.xml
2022-05-18T04:28:02.4842779Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:28:02.4852967Z 
2022-05-18T04:28:02.4853283Z Running tests...
2022-05-18T04:28:02.7679616Z ----------------------------------------------------------------------
2022-05-18T04:28:02.7680495Z   test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22394
2022-05-18T04:28:02.7702473Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22395
2022-05-18T04:28:02.7725557Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22396
2022-05-18T04:28:03.5702476Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:28:03.5803461Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:28:03.5804130Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:28:03.5805128Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:28:03.5805993Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:28:03.5806546Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:28:03.5911231Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:28:03.6818243Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:28:03.6818869Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:28:03.8776011Z skip: CUDA is not available. (1.392s)
2022-05-18T04:28:03.8776287Z 
2022-05-18T04:28:03.8776722Z ----------------------------------------------------------------------
2022-05-18T04:28:03.8777112Z Ran 1 test in 1.392s
2022-05-18T04:28:03.8777289Z 
2022-05-18T04:28:03.8777406Z OK (skipped=1)
2022-05-18T04:28:03.8777578Z 
2022-05-18T04:28:03.8777716Z Generating XML reports...
2022-05-18T04:28:03.8808720Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042802.xml
2022-05-18T04:28:04.7951965Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:28:04.7961457Z 
2022-05-18T04:28:04.7961824Z Running tests...
2022-05-18T04:28:05.0687037Z ----------------------------------------------------------------------
2022-05-18T04:28:05.0688111Z   test_DistributedDataParallel_non_default_stream (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/76428 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure IN_CI is not set and you are not using the flag --import-disabled-tests. (0.272s)
2022-05-18T04:28:05.0688710Z 
2022-05-18T04:28:05.0688911Z ----------------------------------------------------------------------
2022-05-18T04:28:05.0689158Z Ran 1 test in 0.272s
2022-05-18T04:28:05.0689275Z 
2022-05-18T04:28:05.0689349Z OK (skipped=1)
2022-05-18T04:28:05.0689458Z 
2022-05-18T04:28:05.0689544Z Generating XML reports...
2022-05-18T04:28:05.0715611Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042804.xml
2022-05-18T04:28:05.9644581Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:28:05.9654775Z 
2022-05-18T04:28:05.9654903Z Running tests...
2022-05-18T04:28:05.9655508Z ----------------------------------------------------------------------
2022-05-18T04:28:06.2460221Z   test_DistributedDataParallel_requires_grad (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22457
2022-05-18T04:28:06.2482867Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22458
2022-05-18T04:28:06.2505889Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22459
2022-05-18T04:28:07.1196390Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:28:07.1296647Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:28:07.1297529Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:28:07.1298973Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:28:07.1300017Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:28:07.1300994Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:28:07.1405410Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:28:07.2310472Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:28:07.2310926Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:28:07.5560060Z ok (1.590s)
2022-05-18T04:28:07.5560219Z 
2022-05-18T04:28:07.5560589Z ----------------------------------------------------------------------
2022-05-18T04:28:07.5560869Z Ran 1 test in 1.590s
2022-05-18T04:28:07.5560984Z 
2022-05-18T04:28:07.5561033Z OK
2022-05-18T04:28:07.5561124Z 
2022-05-18T04:28:07.5561217Z Generating XML reports...
2022-05-18T04:28:07.5592001Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042805.xml
2022-05-18T04:28:08.4804593Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:28:08.4814810Z 
2022-05-18T04:28:08.4815012Z Running tests...
2022-05-18T04:28:08.4815475Z ----------------------------------------------------------------------
2022-05-18T04:28:08.7556831Z   test_DistributedDataParallel_with_amp_and_grad_is_view (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77294 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure IN_CI is not set and you are not using the flag --import-disabled-tests. (0.274s)
2022-05-18T04:28:08.7557830Z 
2022-05-18T04:28:08.7558211Z ----------------------------------------------------------------------
2022-05-18T04:28:08.7558617Z Ran 1 test in 0.274s
2022-05-18T04:28:08.7558797Z 
2022-05-18T04:28:08.7558899Z OK (skipped=1)
2022-05-18T04:28:08.7559078Z 
2022-05-18T04:28:08.7559218Z Generating XML reports...
2022-05-18T04:28:08.7586583Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042808.xml
2022-05-18T04:28:09.6510348Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:28:09.6520997Z 
2022-05-18T04:28:09.6521318Z Running tests...
2022-05-18T04:28:09.6521968Z ----------------------------------------------------------------------
2022-05-18T04:28:09.9356747Z   test_DistributedSampler_padding (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22520
2022-05-18T04:28:09.9378121Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22521
2022-05-18T04:28:09.9401006Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22522
2022-05-18T04:28:10.7500324Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:28:10.7601638Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:28:10.7602313Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:28:10.7603234Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:28:10.7604068Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:28:10.7604604Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:28:10.7612009Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:28:10.7612776Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:28:10.7614299Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:28:10.9451694Z skip: Need at least 3 CUDA devices (1.293s)
2022-05-18T04:28:10.9451975Z 
2022-05-18T04:28:10.9452472Z ----------------------------------------------------------------------
2022-05-18T04:28:10.9452781Z Ran 1 test in 1.293s
2022-05-18T04:28:10.9452883Z 
2022-05-18T04:28:10.9452957Z OK (skipped=1)
2022-05-18T04:28:10.9453064Z 
2022-05-18T04:28:10.9453148Z Generating XML reports...
2022-05-18T04:28:10.9483754Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042809.xml
2022-05-18T04:28:11.8620679Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:28:11.8630922Z 
2022-05-18T04:28:11.8631036Z Running tests...
2022-05-18T04:28:11.8631500Z ----------------------------------------------------------------------
2022-05-18T04:28:11.8647423Z   test_SyncBatchNorm_process_group (__main__.TestDistBackendWithSpawn) ... skip: no torchvision (0.002s)
2022-05-18T04:28:11.8647922Z 
2022-05-18T04:28:11.8648365Z ----------------------------------------------------------------------
2022-05-18T04:28:11.8648641Z Ran 1 test in 0.002s
2022-05-18T04:28:11.8648761Z 
2022-05-18T04:28:11.8648837Z OK (skipped=1)
2022-05-18T04:28:11.8648948Z 
2022-05-18T04:28:11.8649033Z Generating XML reports...
2022-05-18T04:28:11.8679825Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042811.xml
2022-05-18T04:28:12.6959299Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:28:12.6968990Z 
2022-05-18T04:28:12.6969131Z Running tests...
2022-05-18T04:28:12.6969535Z ----------------------------------------------------------------------
2022-05-18T04:28:12.6983851Z   test_accumulate_gradients_no_sync (__main__.TestDistBackendWithSpawn)
2022-05-18T04:28:12.9763295Z Runs _test_accumulate_gradients_no_sync using default inputs ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22583
2022-05-18T04:28:12.9784902Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22584
2022-05-18T04:28:12.9808049Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22585
2022-05-18T04:28:13.8858253Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:28:13.8959566Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:28:13.8959999Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:28:13.8960642Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:28:13.8961399Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:28:13.8962307Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:28:13.8969893Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:28:13.8970391Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:28:13.8971434Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:28:13.9043340Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp909ycy4b
2022-05-18T04:28:13.9045419Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp909ycy4b/_remote_module_non_scriptable.py
2022-05-18T04:28:13.9075469Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcbbsovka
2022-05-18T04:28:13.9076255Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgnf6ieme
2022-05-18T04:28:13.9077973Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcbbsovka/_remote_module_non_scriptable.py
2022-05-18T04:28:13.9078664Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgnf6ieme/_remote_module_non_scriptable.py
2022-05-18T04:28:14.0857864Z ok (1.389s)
2022-05-18T04:28:14.0858173Z 
2022-05-18T04:28:14.0858645Z ----------------------------------------------------------------------
2022-05-18T04:28:14.0859039Z Ran 1 test in 1.389s
2022-05-18T04:28:14.0859220Z 
2022-05-18T04:28:14.0859317Z OK
2022-05-18T04:28:14.0859451Z 
2022-05-18T04:28:14.0859595Z Generating XML reports...
2022-05-18T04:28:14.0890926Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042812.xml
2022-05-18T04:28:15.0031327Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:28:15.0041895Z 
2022-05-18T04:28:15.0042200Z Running tests...
2022-05-18T04:28:15.0042833Z ----------------------------------------------------------------------
2022-05-18T04:28:15.0060595Z   test_accumulate_gradients_no_sync_allreduce_hook (__main__.TestDistBackendWithSpawn)
2022-05-18T04:28:15.2852897Z Runs multiple iterations on _test_accumulate_gradients_no_sync ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22636
2022-05-18T04:28:15.2874572Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22637
2022-05-18T04:28:15.2897377Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22638
2022-05-18T04:28:16.0802553Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:28:16.0904236Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:28:16.0904794Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:28:16.0905424Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:28:16.0905949Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:28:16.0906474Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:28:16.0914439Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:28:16.0915257Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:28:16.0916096Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:28:16.0989214Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp89mxb516
2022-05-18T04:28:16.0991280Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp89mxb516/_remote_module_non_scriptable.py
2022-05-18T04:28:16.1020298Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyo1g0r3h
2022-05-18T04:28:16.1020720Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw25hnyzb
2022-05-18T04:28:16.1023115Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyo1g0r3h/_remote_module_non_scriptable.py
2022-05-18T04:28:16.1023541Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw25hnyzb/_remote_module_non_scriptable.py
2022-05-18T04:28:16.1275660Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:28:16.1276087Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:28:16.1276631Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:28:16.2947563Z ok (1.290s)
2022-05-18T04:28:16.2947800Z 
2022-05-18T04:28:16.2948380Z ----------------------------------------------------------------------
2022-05-18T04:28:16.2948631Z Ran 1 test in 1.291s
2022-05-18T04:28:16.2948747Z 
2022-05-18T04:28:16.2948811Z OK
2022-05-18T04:28:16.2948908Z 
2022-05-18T04:28:16.2949002Z Generating XML reports...
2022-05-18T04:28:16.2979572Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042815.xml
2022-05-18T04:28:17.2158119Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:28:17.2168508Z 
2022-05-18T04:28:17.2168653Z Running tests...
2022-05-18T04:28:17.2169083Z ----------------------------------------------------------------------
2022-05-18T04:28:17.2189342Z   test_accumulate_gradients_no_sync_allreduce_with_then_hook (__main__.TestDistBackendWithSpawn)
2022-05-18T04:28:17.5038094Z Runs multiple iterations on _test_accumulate_gradients_no_sync using allreduce ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22689
2022-05-18T04:28:17.5060579Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22690
2022-05-18T04:28:17.5084442Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22691
2022-05-18T04:28:18.3195710Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:28:18.3296845Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:28:18.3297528Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:28:18.3298525Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:28:18.3299254Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:28:18.3299815Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:28:18.3307267Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:28:18.3307938Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:28:18.3308740Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:28:18.3378542Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr1275ncb
2022-05-18T04:28:18.3379223Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1vkt9qlo
2022-05-18T04:28:18.3379844Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr1275ncb/_remote_module_non_scriptable.py
2022-05-18T04:28:18.3380820Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptebugg8v
2022-05-18T04:28:18.3381539Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1vkt9qlo/_remote_module_non_scriptable.py
2022-05-18T04:28:18.3383110Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptebugg8v/_remote_module_non_scriptable.py
2022-05-18T04:28:18.3567795Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:28:18.3568539Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:28:18.3569236Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:28:18.5133949Z ok (1.296s)
2022-05-18T04:28:18.5134237Z 
2022-05-18T04:28:18.5134700Z ----------------------------------------------------------------------
2022-05-18T04:28:18.5135092Z Ran 1 test in 1.296s
2022-05-18T04:28:18.5135278Z 
2022-05-18T04:28:18.5135378Z OK
2022-05-18T04:28:18.5135817Z 
2022-05-18T04:28:18.5135969Z Generating XML reports...
2022-05-18T04:28:18.5167476Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042817.xml
2022-05-18T04:28:19.4313771Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:28:19.4323752Z 
2022-05-18T04:28:19.4323842Z Running tests...
2022-05-18T04:28:19.4324650Z ----------------------------------------------------------------------
2022-05-18T04:28:19.4338727Z   test_accumulate_gradients_no_sync_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-05-18T04:28:19.7140702Z Runs _test_accumulate_gradients_no_sync using default inputs ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22742
2022-05-18T04:28:19.7163790Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22743
2022-05-18T04:28:19.7186470Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22744
2022-05-18T04:28:20.5187944Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:28:20.5289238Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:28:20.5289830Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:28:20.5290470Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:28:20.5290983Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:28:20.5291588Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:28:20.5397059Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:28:20.5466205Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptcl2negt
2022-05-18T04:28:20.5467938Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptcl2negt/_remote_module_non_scriptable.py
2022-05-18T04:28:20.6303278Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:28:20.6304005Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:28:20.6409984Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmple55gk83
2022-05-18T04:28:20.6410430Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsj87y_l_
2022-05-18T04:28:20.6412564Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmple55gk83/_remote_module_non_scriptable.py
2022-05-18T04:28:20.6413033Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsj87y_l_/_remote_module_non_scriptable.py
2022-05-18T04:28:20.8238318Z ok (1.391s)
2022-05-18T04:28:20.8238565Z 
2022-05-18T04:28:20.8239010Z ----------------------------------------------------------------------
2022-05-18T04:28:20.8239264Z Ran 1 test in 1.391s
2022-05-18T04:28:20.8239382Z 
2022-05-18T04:28:20.8239440Z OK
2022-05-18T04:28:20.8239600Z 
2022-05-18T04:28:20.8239701Z Generating XML reports...
2022-05-18T04:28:20.8270571Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042819.xml
2022-05-18T04:28:21.7455106Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:28:21.7465137Z 
2022-05-18T04:28:21.7465305Z Running tests...
2022-05-18T04:28:21.7465648Z ----------------------------------------------------------------------
2022-05-18T04:28:22.0260116Z   test_all_gather (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22795
2022-05-18T04:28:22.0281527Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22796
2022-05-18T04:28:22.0304703Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22797
2022-05-18T04:28:22.8365801Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:28:22.8466785Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:28:22.8467242Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:28:22.8467861Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:28:22.8468697Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:28:22.8469564Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:28:22.8476501Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:28:22.8477632Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:28:22.8478296Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:28:23.1355413Z ok (1.389s)
2022-05-18T04:28:23.1355662Z 
2022-05-18T04:28:23.1356163Z ----------------------------------------------------------------------
2022-05-18T04:28:23.1356525Z Ran 1 test in 1.389s
2022-05-18T04:28:23.1356706Z 
2022-05-18T04:28:23.1356805Z OK
2022-05-18T04:28:23.1356957Z 
2022-05-18T04:28:23.1357093Z Generating XML reports...
2022-05-18T04:28:23.1388170Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042821.xml
2022-05-18T04:28:24.0580043Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:28:24.0589693Z 
2022-05-18T04:28:24.0589956Z Running tests...
2022-05-18T04:28:24.0590330Z ----------------------------------------------------------------------
2022-05-18T04:28:24.3372258Z   test_all_gather_coalesced_complex (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22851
2022-05-18T04:28:24.3395207Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22852
2022-05-18T04:28:24.3418285Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22853
2022-05-18T04:28:25.1177026Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:28:25.1277516Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:28:25.1278151Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:28:25.1278774Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:28:25.1279322Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:28:25.1279848Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:28:25.1385367Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:28:25.2290183Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:28:25.2290583Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:28:25.5470122Z ok (1.488s)
2022-05-18T04:28:25.5470367Z 
2022-05-18T04:28:25.5470801Z ----------------------------------------------------------------------
2022-05-18T04:28:25.5471193Z Ran 1 test in 1.488s
2022-05-18T04:28:25.5471380Z 
2022-05-18T04:28:25.5471476Z OK
2022-05-18T04:28:25.5471618Z 
2022-05-18T04:28:25.5471756Z Generating XML reports...
2022-05-18T04:28:25.5503253Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042824.xml
2022-05-18T04:28:26.4681807Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:28:26.4691523Z 
2022-05-18T04:28:26.4691648Z Running tests...
2022-05-18T04:28:26.4692457Z ----------------------------------------------------------------------
2022-05-18T04:28:26.7516406Z   test_all_gather_coalesced_full_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22907
2022-05-18T04:28:26.7538437Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22908
2022-05-18T04:28:26.7561964Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22909
2022-05-18T04:28:27.5738596Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:28:27.5840351Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:28:27.5841614Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:28:27.5842024Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:28:27.5842531Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:28:27.5843049Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:28:27.5850279Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:28:27.5850714Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:28:27.5853182Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:28:27.5959717Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:28:27.6060728Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-05-18T04:28:27.6061307Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:28:27.6062224Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:28:27.6063269Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:28:27.6063967Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:28:27.8611587Z ok (1.392s)
2022-05-18T04:28:27.8612032Z 
2022-05-18T04:28:27.8612728Z ----------------------------------------------------------------------
2022-05-18T04:28:27.8613121Z Ran 1 test in 1.392s
2022-05-18T04:28:27.8613242Z 
2022-05-18T04:28:27.8613310Z OK
2022-05-18T04:28:27.8613403Z 
2022-05-18T04:28:27.8613499Z Generating XML reports...
2022-05-18T04:28:27.8644722Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042826.xml
2022-05-18T04:28:28.7891734Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:28:28.7902377Z 
2022-05-18T04:28:28.7902817Z Running tests...
2022-05-18T04:28:28.7903661Z ----------------------------------------------------------------------
2022-05-18T04:28:29.0719776Z   test_all_gather_coalesced_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22972
2022-05-18T04:28:29.0741225Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22973
2022-05-18T04:28:29.0763772Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22974
2022-05-18T04:28:29.8533924Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:28:29.8630358Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:28:29.8630775Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:28:29.8631383Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:28:29.8631918Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:28:29.8634675Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:28:29.8739042Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:28:29.8739524Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:28:29.8844669Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:28:29.8845105Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-05-18T04:28:29.9645000Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:28:29.9647693Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:28:29.9648337Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:28:29.9653169Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:28:29.9653857Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:28:30.2815860Z ok (1.491s)
2022-05-18T04:28:30.2816084Z 
2022-05-18T04:28:30.2816554Z ----------------------------------------------------------------------
2022-05-18T04:28:30.2816976Z Ran 1 test in 1.491s
2022-05-18T04:28:30.2817180Z 
2022-05-18T04:28:30.2817273Z OK
2022-05-18T04:28:30.2817406Z 
2022-05-18T04:28:30.2817543Z Generating XML reports...
2022-05-18T04:28:30.2848291Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042828.xml
2022-05-18T04:28:31.1964737Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:28:31.1974315Z 
2022-05-18T04:28:31.1974439Z Running tests...
2022-05-18T04:28:31.1974884Z ----------------------------------------------------------------------
2022-05-18T04:28:31.4773797Z   test_all_gather_coalesced_simple (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23033
2022-05-18T04:28:31.4795807Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23034
2022-05-18T04:28:31.4819937Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23035
2022-05-18T04:28:32.2980821Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:28:32.2981572Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:28:32.2982020Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:28:32.2982639Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:28:32.2983344Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:28:32.2983868Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:28:32.2992584Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:28:32.2993463Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:28:32.2994108Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:28:32.5871096Z ok (1.389s)
2022-05-18T04:28:32.5871357Z 
2022-05-18T04:28:32.5871864Z ----------------------------------------------------------------------
2022-05-18T04:28:32.5872152Z Ran 1 test in 1.390s
2022-05-18T04:28:32.5872283Z 
2022-05-18T04:28:32.5872345Z OK
2022-05-18T04:28:32.5872441Z 
2022-05-18T04:28:32.5872534Z Generating XML reports...
2022-05-18T04:28:32.5903254Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042831.xml
2022-05-18T04:28:33.5130966Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:28:33.5141023Z 
2022-05-18T04:28:33.5141374Z Running tests...
2022-05-18T04:28:33.5141821Z ----------------------------------------------------------------------
2022-05-18T04:28:33.7966234Z   test_all_gather_coalesced_with_empty (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23089
2022-05-18T04:28:33.7988071Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23090
2022-05-18T04:28:33.8010714Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23091
2022-05-18T04:28:34.5985639Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:28:34.6040931Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:28:34.6041566Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:28:34.6042189Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:28:34.6042731Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:28:34.6086770Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:28:34.6149737Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:28:34.6150544Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:28:34.7097759Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:28:35.0065688Z ok (1.492s)
2022-05-18T04:28:35.0065925Z 
2022-05-18T04:28:35.0066372Z ----------------------------------------------------------------------
2022-05-18T04:28:35.0066771Z Ran 1 test in 1.492s
2022-05-18T04:28:35.0066964Z 
2022-05-18T04:28:35.0067056Z OK
2022-05-18T04:28:35.0067199Z 
2022-05-18T04:28:35.0067642Z Generating XML reports...
2022-05-18T04:28:35.0098989Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042833.xml
2022-05-18T04:28:35.9264669Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:28:35.9274643Z 
2022-05-18T04:28:35.9274731Z Running tests...
2022-05-18T04:28:35.9275554Z ----------------------------------------------------------------------
2022-05-18T04:28:36.2081325Z   test_all_gather_complex (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23145
2022-05-18T04:28:36.2103213Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23146
2022-05-18T04:28:36.2126115Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23147
2022-05-18T04:28:37.0347368Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:28:37.0447661Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:28:37.0448334Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:28:37.0449344Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:28:37.0449873Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:28:37.0450400Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:28:37.0556622Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:28:37.0557967Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:28:37.1460613Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:28:37.4177504Z ok (1.490s)
2022-05-18T04:28:37.4177776Z 
2022-05-18T04:28:37.4178289Z ----------------------------------------------------------------------
2022-05-18T04:28:37.4178544Z Ran 1 test in 1.490s
2022-05-18T04:28:37.4178659Z 
2022-05-18T04:28:37.4178723Z OK
2022-05-18T04:28:37.4178815Z 
2022-05-18T04:28:37.4178896Z Generating XML reports...
2022-05-18T04:28:37.4209730Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042835.xml
2022-05-18T04:28:38.3376954Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:28:38.3386262Z 
2022-05-18T04:28:38.3386366Z Running tests...
2022-05-18T04:28:38.3386818Z ----------------------------------------------------------------------
2022-05-18T04:28:38.3403440Z   test_all_gather_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all gather (0.002s)
2022-05-18T04:28:38.3403738Z 
2022-05-18T04:28:38.3404010Z ----------------------------------------------------------------------
2022-05-18T04:28:38.3404262Z Ran 1 test in 0.002s
2022-05-18T04:28:38.3404379Z 
2022-05-18T04:28:38.3404453Z OK (skipped=1)
2022-05-18T04:28:38.3404559Z 
2022-05-18T04:28:38.3404645Z Generating XML reports...
2022-05-18T04:28:38.3436938Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042838.xml
2022-05-18T04:28:39.1759285Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:28:39.1769838Z 
2022-05-18T04:28:39.1770124Z Running tests...
2022-05-18T04:28:39.1770542Z ----------------------------------------------------------------------
2022-05-18T04:28:39.1786661Z   test_all_gather_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all gather (0.002s)
2022-05-18T04:28:39.1787074Z 
2022-05-18T04:28:39.1787446Z ----------------------------------------------------------------------
2022-05-18T04:28:39.1787897Z Ran 1 test in 0.002s
2022-05-18T04:28:39.1788058Z 
2022-05-18T04:28:39.1788139Z OK (skipped=1)
2022-05-18T04:28:39.1788238Z 
2022-05-18T04:28:39.1788324Z Generating XML reports...
2022-05-18T04:28:39.1819200Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042839.xml
2022-05-18T04:28:40.0218573Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:28:40.0228402Z 
2022-05-18T04:28:40.0228602Z Running tests...
2022-05-18T04:28:40.0229698Z ----------------------------------------------------------------------
2022-05-18T04:28:40.3061344Z   test_all_gather_full_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23221
2022-05-18T04:28:40.3083707Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23222
2022-05-18T04:28:40.3106570Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23223
2022-05-18T04:28:41.1137279Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:28:41.1137762Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:28:41.1138116Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:28:41.1138730Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:28:41.1139260Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:28:41.1139771Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:28:41.1243003Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:28:41.2148387Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:28:41.2148804Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:28:41.2354177Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:28:41.2456023Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-05-18T04:28:41.2456662Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:28:41.2457656Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:28:41.2458203Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:28:41.2458721Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:28:41.5157626Z ok (1.493s)
2022-05-18T04:28:41.5157872Z 
2022-05-18T04:28:41.5158174Z ----------------------------------------------------------------------
2022-05-18T04:28:41.5158432Z Ran 1 test in 1.493s
2022-05-18T04:28:41.5158583Z 
2022-05-18T04:28:41.5158669Z OK
2022-05-18T04:28:41.5158760Z 
2022-05-18T04:28:41.5158842Z Generating XML reports...
2022-05-18T04:28:41.5189039Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042840.xml
2022-05-18T04:28:42.4465818Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:28:42.4466121Z 
2022-05-18T04:28:42.4466244Z Running tests...
2022-05-18T04:28:42.4466597Z ----------------------------------------------------------------------
2022-05-18T04:28:42.7263322Z   test_all_gather_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23286
2022-05-18T04:28:42.7285835Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23287
2022-05-18T04:28:42.7308597Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23288
2022-05-18T04:28:43.5373726Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:28:43.5475285Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:28:43.5476051Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:28:43.5476785Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:28:43.5477318Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:28:43.5478013Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:28:43.5485475Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:28:43.5486036Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:28:43.5486968Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:28:43.5489865Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:28:43.5692157Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-05-18T04:28:43.5692859Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:28:43.5693920Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:28:43.5694531Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:28:43.5695063Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:28:43.9361339Z ok (1.489s)
2022-05-18T04:28:43.9361585Z 
2022-05-18T04:28:43.9362240Z ----------------------------------------------------------------------
2022-05-18T04:28:43.9362634Z Ran 1 test in 1.489s
2022-05-18T04:28:43.9362737Z 
2022-05-18T04:28:43.9362799Z OK
2022-05-18T04:28:43.9362892Z 
2022-05-18T04:28:43.9362986Z Generating XML reports...
2022-05-18T04:28:43.9393159Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042842.xml
2022-05-18T04:28:44.8619477Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:28:44.8630427Z 
2022-05-18T04:28:44.8630756Z Running tests...
2022-05-18T04:28:44.8631408Z ----------------------------------------------------------------------
2022-05-18T04:28:44.8647503Z   test_all_gather_multigpu (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl backend supports allgather multigpu (0.002s)
2022-05-18T04:28:44.8647829Z 
2022-05-18T04:28:44.8648215Z ----------------------------------------------------------------------
2022-05-18T04:28:44.8648676Z Ran 1 test in 0.002s
2022-05-18T04:28:44.8648874Z 
2022-05-18T04:28:44.8648990Z OK (skipped=1)
2022-05-18T04:28:44.8649099Z 
2022-05-18T04:28:44.8649186Z Generating XML reports...
2022-05-18T04:28:44.8679606Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042844.xml
2022-05-18T04:28:45.6970581Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:28:45.6981481Z 
2022-05-18T04:28:45.6981793Z Running tests...
2022-05-18T04:28:45.6982473Z ----------------------------------------------------------------------
2022-05-18T04:28:45.6999569Z   test_all_gather_multigpu_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl backend supports allgather multigpu (0.002s)
2022-05-18T04:28:45.7000015Z 
2022-05-18T04:28:45.7000398Z ----------------------------------------------------------------------
2022-05-18T04:28:45.7000799Z Ran 1 test in 0.002s
2022-05-18T04:28:45.7000989Z 
2022-05-18T04:28:45.7001106Z OK (skipped=1)
2022-05-18T04:28:45.7001289Z 
2022-05-18T04:28:45.7001426Z Generating XML reports...
2022-05-18T04:28:45.7033024Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042845.xml
2022-05-18T04:28:46.5337099Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:28:46.5347612Z 
2022-05-18T04:28:46.5347894Z Running tests...
2022-05-18T04:28:46.5348492Z ----------------------------------------------------------------------
2022-05-18T04:28:46.8140097Z   test_all_gather_object_default_pg (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23367
2022-05-18T04:28:46.8163431Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23368
2022-05-18T04:28:46.8186494Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23369
2022-05-18T04:28:47.6501524Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:28:47.6602872Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:28:47.6603737Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:28:47.6604134Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:28:47.6604670Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:28:47.6605215Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:28:47.6612925Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:28:47.6613407Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:28:47.6615045Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:28:47.8235139Z ok (1.288s)
2022-05-18T04:28:47.8235427Z 
2022-05-18T04:28:47.8235915Z ----------------------------------------------------------------------
2022-05-18T04:28:47.8236302Z Ran 1 test in 1.289s
2022-05-18T04:28:47.8236480Z 
2022-05-18T04:28:47.8236575Z OK
2022-05-18T04:28:47.8236710Z 
2022-05-18T04:28:47.8236859Z Generating XML reports...
2022-05-18T04:28:47.8267462Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042846.xml
2022-05-18T04:28:48.7419279Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:28:48.7428914Z 
2022-05-18T04:28:48.7429060Z Running tests...
2022-05-18T04:28:48.7429502Z ----------------------------------------------------------------------
2022-05-18T04:28:49.0221153Z   test_all_gather_object_subgroup (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23420
2022-05-18T04:28:49.0243831Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23421
2022-05-18T04:28:49.0266849Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23422
2022-05-18T04:28:49.8477297Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:28:49.8577956Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:28:49.8578946Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:28:49.8579646Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:28:49.8580483Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:28:49.8581052Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:28:49.8588765Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:28:49.8590119Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:28:49.8590775Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:28:49.9103103Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:28:49.9203396Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-05-18T04:28:49.9204172Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:28:49.9205233Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:28:49.9205932Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:28:49.9206442Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:28:49.9548239Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-05-18T04:28:49.9548682Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-05-18T04:28:49.9549143Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 2
2022-05-18T04:28:49.9549810Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 3 nodes.
2022-05-18T04:28:49.9550563Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 3 nodes.
2022-05-18T04:28:49.9551132Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:3 with 3 nodes.
2022-05-18T04:28:49.9670346Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 0
2022-05-18T04:28:49.9771827Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 1
2022-05-18T04:28:49.9772426Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 2
2022-05-18T04:28:49.9773320Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:4 with 3 nodes.
2022-05-18T04:28:49.9773898Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:4 with 3 nodes.
2022-05-18T04:28:49.9774408Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:4 with 3 nodes.
2022-05-18T04:28:50.2318787Z ok (1.489s)
2022-05-18T04:28:50.2318994Z 
2022-05-18T04:28:50.2319440Z ----------------------------------------------------------------------
2022-05-18T04:28:50.2319844Z Ran 1 test in 1.489s
2022-05-18T04:28:50.2320061Z 
2022-05-18T04:28:50.2320157Z OK
2022-05-18T04:28:50.2320337Z 
2022-05-18T04:28:50.2320487Z Generating XML reports...
2022-05-18T04:28:50.2352141Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042848.xml
2022-05-18T04:28:51.1630949Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:28:51.1641028Z 
2022-05-18T04:28:51.1641223Z Running tests...
2022-05-18T04:28:51.1641624Z ----------------------------------------------------------------------
2022-05-18T04:28:51.4467374Z   test_all_reduce_coalesced_full_group_max (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23509
2022-05-18T04:28:51.4488668Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23510
2022-05-18T04:28:51.4512061Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23511
2022-05-18T04:28:52.2877646Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:28:52.2978878Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:28:52.2979606Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:28:52.2980464Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:28:52.2981067Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:28:52.2981598Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:28:52.3089954Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:28:52.3090372Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:28:52.3992205Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:28:52.4101860Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:28:52.4203809Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-05-18T04:28:52.4204314Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:28:52.4205199Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:28:52.4205940Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:28:52.4206479Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:28:52.6563671Z ok (1.492s)
2022-05-18T04:28:52.6563884Z 
2022-05-18T04:28:52.6564198Z ----------------------------------------------------------------------
2022-05-18T04:28:52.6564455Z Ran 1 test in 1.492s
2022-05-18T04:28:52.6564574Z 
2022-05-18T04:28:52.6564637Z OK
2022-05-18T04:28:52.6564728Z 
2022-05-18T04:28:52.6564823Z Generating XML reports...
2022-05-18T04:28:52.6595382Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042851.xml
2022-05-18T04:28:53.5932734Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:28:53.5942831Z 
2022-05-18T04:28:53.5943327Z Running tests...
2022-05-18T04:28:53.5943680Z ----------------------------------------------------------------------
2022-05-18T04:28:53.8772269Z   test_all_reduce_coalesced_full_group_min (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23574
2022-05-18T04:28:53.8794550Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23575
2022-05-18T04:28:53.8817979Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23576
2022-05-18T04:28:54.7171535Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:28:54.7172142Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:28:54.7172520Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:28:54.7173132Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:28:54.7173665Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:28:54.7174191Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:28:54.7280432Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:28:54.7280956Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:28:54.8183577Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:28:54.8290565Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:28:54.8393055Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:28:54.8393954Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:28:54.8394580Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-05-18T04:28:54.8395522Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:28:54.8396278Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:28:55.0870436Z ok (1.492s)
2022-05-18T04:28:55.0870747Z 
2022-05-18T04:28:55.0871238Z ----------------------------------------------------------------------
2022-05-18T04:28:55.0871534Z Ran 1 test in 1.493s
2022-05-18T04:28:55.0871651Z 
2022-05-18T04:28:55.0871712Z OK
2022-05-18T04:28:55.0871805Z 
2022-05-18T04:28:55.0871896Z Generating XML reports...
2022-05-18T04:28:55.0904426Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042853.xml
2022-05-18T04:28:56.0051893Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:28:56.0062503Z 
2022-05-18T04:28:56.0063108Z Running tests...
2022-05-18T04:28:56.0063628Z ----------------------------------------------------------------------
2022-05-18T04:28:56.2885919Z   test_all_reduce_coalesced_full_group_product (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23639
2022-05-18T04:28:56.2908167Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23640
2022-05-18T04:28:56.2931164Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23641
2022-05-18T04:28:57.0977543Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:28:57.1066594Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:28:57.1067043Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:28:57.1067647Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:28:57.1068175Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:28:57.1078493Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:28:57.1176067Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:28:57.1176669Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:28:57.2088437Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:28:57.2287257Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:28:57.2387840Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:28:57.2388302Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-05-18T04:28:57.2389043Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:28:57.2389655Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:28:57.2390181Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:28:57.5985830Z ok (1.592s)
2022-05-18T04:28:57.5986077Z 
2022-05-18T04:28:57.5986571Z ----------------------------------------------------------------------
2022-05-18T04:28:57.5986862Z Ran 1 test in 1.592s
2022-05-18T04:28:57.5986986Z 
2022-05-18T04:28:57.5987054Z OK
2022-05-18T04:28:57.5987146Z 
2022-05-18T04:28:57.5987238Z Generating XML reports...
2022-05-18T04:28:57.6017279Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042856.xml
2022-05-18T04:28:58.5332086Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:28:58.5342312Z 
2022-05-18T04:28:58.5342603Z Running tests...
2022-05-18T04:28:58.5343426Z ----------------------------------------------------------------------
2022-05-18T04:28:58.8201868Z   test_all_reduce_coalesced_full_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23704
2022-05-18T04:28:58.8224447Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23705
2022-05-18T04:28:58.8247282Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23706
2022-05-18T04:28:59.7058596Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:28:59.7066531Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:28:59.7067038Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:28:59.7067718Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:28:59.7068240Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:28:59.7159939Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:28:59.7174868Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:28:59.7175928Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:28:59.8171984Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:28:59.8479437Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:28:59.8580651Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:28:59.8581270Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-05-18T04:28:59.8582139Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:28:59.8583135Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:28:59.8584057Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:29:00.1301411Z ok (1.596s)
2022-05-18T04:29:00.1301651Z 
2022-05-18T04:29:00.1302182Z ----------------------------------------------------------------------
2022-05-18T04:29:00.1302541Z Ran 1 test in 1.596s
2022-05-18T04:29:00.1302656Z 
2022-05-18T04:29:00.1302720Z OK
2022-05-18T04:29:00.1302810Z 
2022-05-18T04:29:00.1303026Z Generating XML reports...
2022-05-18T04:29:00.1333474Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042858.xml
2022-05-18T04:29:01.0627185Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:29:01.0636103Z 
2022-05-18T04:29:01.0636317Z Running tests...
2022-05-18T04:29:01.0636639Z ----------------------------------------------------------------------
2022-05-18T04:29:01.3447078Z   test_all_reduce_coalesced_group_max (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23769
2022-05-18T04:29:01.3469877Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23770
2022-05-18T04:29:01.3492483Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23771
2022-05-18T04:29:02.1529093Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:29:02.1630263Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:29:02.1630699Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:29:02.1631344Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:29:02.1631903Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:29:02.1632419Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:29:02.1640593Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:29:02.1641177Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:29:02.1642274Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:29:02.1642838Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:29:02.1848782Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:29:02.1849219Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-05-18T04:29:02.1849894Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:29:02.1850458Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:29:02.1947581Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:29:02.5545429Z ok (1.491s)
2022-05-18T04:29:02.5545636Z 
2022-05-18T04:29:02.5546151Z ----------------------------------------------------------------------
2022-05-18T04:29:02.5546623Z Ran 1 test in 1.491s
2022-05-18T04:29:02.5546781Z 
2022-05-18T04:29:02.5546841Z OK
2022-05-18T04:29:02.5546933Z 
2022-05-18T04:29:02.5547015Z Generating XML reports...
2022-05-18T04:29:02.5579704Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042901.xml
2022-05-18T04:29:03.5075152Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:29:03.5084719Z 
2022-05-18T04:29:03.5084828Z Running tests...
2022-05-18T04:29:03.5085305Z ----------------------------------------------------------------------
2022-05-18T04:29:03.7950282Z   test_all_reduce_coalesced_group_min (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23830
2022-05-18T04:29:03.7972646Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23831
2022-05-18T04:29:03.7995496Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23832
2022-05-18T04:29:04.6402810Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:29:04.6502150Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:29:04.6503100Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:29:04.6503966Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:29:04.6504568Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:29:04.6505084Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:29:04.6609785Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:29:04.7516176Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:29:04.7516797Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:29:04.7517532Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:29:04.7620521Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:29:04.7621243Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-05-18T04:29:04.7622095Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:29:04.7622869Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:29:04.7721399Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:29:05.1048850Z ok (1.596s)
2022-05-18T04:29:05.1049018Z 
2022-05-18T04:29:05.1049419Z ----------------------------------------------------------------------
2022-05-18T04:29:05.1049681Z Ran 1 test in 1.596s
2022-05-18T04:29:05.1049799Z 
2022-05-18T04:29:05.1049862Z OK
2022-05-18T04:29:05.1050008Z 
2022-05-18T04:29:05.1050107Z Generating XML reports...
2022-05-18T04:29:05.1080975Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042903.xml
2022-05-18T04:29:06.0358844Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:29:06.0368991Z 
2022-05-18T04:29:06.0369124Z Running tests...
2022-05-18T04:29:06.0369717Z ----------------------------------------------------------------------
2022-05-18T04:29:06.3162281Z   test_all_reduce_coalesced_group_product (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23891
2022-05-18T04:29:06.3185168Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23892
2022-05-18T04:29:06.3208607Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23893
2022-05-18T04:29:07.1199271Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:29:07.1300738Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:29:07.1301142Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:29:07.1301756Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:29:07.1302288Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:29:07.1302793Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:29:07.1407539Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:29:07.2315904Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:29:07.2316496Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:29:07.2317982Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:29:07.2520672Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-05-18T04:29:07.2521245Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:29:07.2522160Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:29:07.2523062Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:29:07.2523746Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:29:07.5260020Z ok (1.489s)
2022-05-18T04:29:07.5260219Z 
2022-05-18T04:29:07.5260623Z ----------------------------------------------------------------------
2022-05-18T04:29:07.5260974Z Ran 1 test in 1.489s
2022-05-18T04:29:07.5261115Z 
2022-05-18T04:29:07.5261202Z OK
2022-05-18T04:29:07.5261282Z 
2022-05-18T04:29:07.5261388Z Generating XML reports...
2022-05-18T04:29:07.5292229Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042906.xml
2022-05-18T04:29:08.4519402Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:29:08.4529571Z 
2022-05-18T04:29:08.4529685Z Running tests...
2022-05-18T04:29:08.4530105Z ----------------------------------------------------------------------
2022-05-18T04:29:08.7411375Z   test_all_reduce_coalesced_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23952
2022-05-18T04:29:08.7434136Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23953
2022-05-18T04:29:08.7457601Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23954
2022-05-18T04:29:09.5696621Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:29:09.5739254Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:29:09.5739966Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:29:09.5740634Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:29:09.5741165Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:29:09.5797674Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:29:09.5848445Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:29:09.5849240Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:29:09.6053730Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:29:09.6054145Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-05-18T04:29:09.6808138Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:29:09.6810493Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:29:09.6811357Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:29:09.6862523Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:29:09.6863264Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:29:09.9508129Z ok (1.498s)
2022-05-18T04:29:09.9508337Z 
2022-05-18T04:29:09.9508989Z ----------------------------------------------------------------------
2022-05-18T04:29:09.9509345Z Ran 1 test in 1.498s
2022-05-18T04:29:09.9509462Z 
2022-05-18T04:29:09.9509525Z OK
2022-05-18T04:29:09.9509617Z 
2022-05-18T04:29:09.9509714Z Generating XML reports...
2022-05-18T04:29:09.9539771Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042908.xml
2022-05-18T04:29:10.8931972Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:29:10.8942103Z 
2022-05-18T04:29:10.8942208Z Running tests...
2022-05-18T04:29:10.8942767Z ----------------------------------------------------------------------
2022-05-18T04:29:11.1780221Z   test_all_reduce_coalesced_max (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24013
2022-05-18T04:29:11.1802203Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24014
2022-05-18T04:29:11.1825101Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24015
2022-05-18T04:29:12.0575837Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:29:12.0676291Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:29:12.0676707Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:29:12.0677331Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:29:12.0677931Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:29:12.0678460Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:29:12.0785625Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:29:12.0786327Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:29:12.1689995Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:29:12.4880352Z ok (1.593s)
2022-05-18T04:29:12.4880513Z 
2022-05-18T04:29:12.4880819Z ----------------------------------------------------------------------
2022-05-18T04:29:12.4881106Z Ran 1 test in 1.594s
2022-05-18T04:29:12.4881226Z 
2022-05-18T04:29:12.4881293Z OK
2022-05-18T04:29:12.4881421Z 
2022-05-18T04:29:12.4881562Z Generating XML reports...
2022-05-18T04:29:12.4912078Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042910.xml
2022-05-18T04:29:13.4136734Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:29:13.4146515Z 
2022-05-18T04:29:13.4147211Z Running tests...
2022-05-18T04:29:13.4147645Z ----------------------------------------------------------------------
2022-05-18T04:29:13.6958969Z   test_all_reduce_coalesced_max_complex_unsupported (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24069
2022-05-18T04:29:13.6981441Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24070
2022-05-18T04:29:13.7004655Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24071
2022-05-18T04:29:14.4921625Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:29:14.5007588Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:29:14.5008003Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:29:14.5008817Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:29:14.5009399Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:29:14.5023206Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:29:14.5117018Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:29:14.5117533Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:29:14.6033479Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:29:14.8056811Z ok (1.391s)
2022-05-18T04:29:14.8057120Z 
2022-05-18T04:29:14.8057629Z ----------------------------------------------------------------------
2022-05-18T04:29:14.8057905Z Ran 1 test in 1.391s
2022-05-18T04:29:14.8058020Z 
2022-05-18T04:29:14.8058083Z OK
2022-05-18T04:29:14.8058161Z 
2022-05-18T04:29:14.8058258Z Generating XML reports...
2022-05-18T04:29:14.8088410Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042913.xml
2022-05-18T04:29:15.7293649Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:29:15.7303242Z 
2022-05-18T04:29:15.7303483Z Running tests...
2022-05-18T04:29:15.7304196Z ----------------------------------------------------------------------
2022-05-18T04:29:16.0117955Z   test_all_reduce_coalesced_min (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24122
2022-05-18T04:29:16.0140087Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24123
2022-05-18T04:29:16.0163034Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24124
2022-05-18T04:29:16.7988435Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:29:16.8089257Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:29:16.8089956Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:29:16.8090909Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:29:16.8091451Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:29:16.8091973Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:29:16.8100519Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:29:16.8101514Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:29:16.8102401Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:29:17.1216221Z ok (1.391s)
2022-05-18T04:29:17.1216494Z 
2022-05-18T04:29:17.1217036Z ----------------------------------------------------------------------
2022-05-18T04:29:17.1217365Z Ran 1 test in 1.391s
2022-05-18T04:29:17.1217481Z 
2022-05-18T04:29:17.1217543Z OK
2022-05-18T04:29:17.1217635Z 
2022-05-18T04:29:17.1217731Z Generating XML reports...
2022-05-18T04:29:17.1248029Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042915.xml
2022-05-18T04:29:18.0443710Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:29:18.0454263Z 
2022-05-18T04:29:18.0454714Z Running tests...
2022-05-18T04:29:18.0455121Z ----------------------------------------------------------------------
2022-05-18T04:29:18.3250973Z   test_all_reduce_coalesced_product (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24178
2022-05-18T04:29:18.3273762Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24179
2022-05-18T04:29:18.3296396Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24180
2022-05-18T04:29:19.1666959Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:29:19.1747960Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:29:19.1748366Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:29:19.1748978Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:29:19.1749508Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:29:19.1767699Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:29:19.1856944Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:29:19.1857694Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:29:19.2779931Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:29:19.5348628Z ok (1.489s)
2022-05-18T04:29:19.5348802Z 
2022-05-18T04:29:19.5349235Z ----------------------------------------------------------------------
2022-05-18T04:29:19.5349480Z Ran 1 test in 1.489s
2022-05-18T04:29:19.5349598Z 
2022-05-18T04:29:19.5349660Z OK
2022-05-18T04:29:19.5349751Z 
2022-05-18T04:29:19.5349852Z Generating XML reports...
2022-05-18T04:29:19.5379394Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042918.xml
2022-05-18T04:29:20.4562136Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:29:20.4572565Z 
2022-05-18T04:29:20.4572964Z Running tests...
2022-05-18T04:29:20.4573377Z ----------------------------------------------------------------------
2022-05-18T04:29:20.7381065Z   test_all_reduce_coalesced_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24234
2022-05-18T04:29:20.7403629Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24235
2022-05-18T04:29:20.7426403Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24236
2022-05-18T04:29:21.5464509Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:29:21.5564729Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:29:21.5565268Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:29:21.5566188Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:29:21.5566721Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:29:21.5567239Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:29:21.5672651Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:29:21.6577857Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:29:21.6578298Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:29:21.9479145Z ok (1.490s)
2022-05-18T04:29:21.9479729Z 
2022-05-18T04:29:21.9480224Z ----------------------------------------------------------------------
2022-05-18T04:29:21.9480507Z Ran 1 test in 1.491s
2022-05-18T04:29:21.9480623Z 
2022-05-18T04:29:21.9480684Z OK
2022-05-18T04:29:21.9480854Z 
2022-05-18T04:29:21.9480949Z Generating XML reports...
2022-05-18T04:29:21.9512390Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042920.xml
2022-05-18T04:29:22.8800465Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:29:22.8810352Z 
2022-05-18T04:29:22.8810495Z Running tests...
2022-05-18T04:29:22.8811105Z ----------------------------------------------------------------------
2022-05-18T04:29:23.1651079Z   test_all_reduce_complex_unsupported_ops (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24290
2022-05-18T04:29:23.1673583Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24291
2022-05-18T04:29:23.1697076Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24292
2022-05-18T04:29:23.9758885Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:29:23.9859806Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:29:23.9860233Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:29:23.9860855Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:29:23.9861384Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:29:23.9861910Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:29:23.9870997Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:29:23.9871526Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:29:23.9872600Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:29:24.1746277Z ok (1.293s)
2022-05-18T04:29:24.1746442Z 
2022-05-18T04:29:24.1747245Z ----------------------------------------------------------------------
2022-05-18T04:29:24.1747523Z Ran 1 test in 1.294s
2022-05-18T04:29:24.1747644Z 
2022-05-18T04:29:24.1747708Z OK
2022-05-18T04:29:24.1747843Z 
2022-05-18T04:29:24.1747949Z Generating XML reports...
2022-05-18T04:29:24.1778217Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042922.xml
2022-05-18T04:29:25.0987996Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:29:25.0998135Z 
2022-05-18T04:29:25.0998267Z Running tests...
2022-05-18T04:29:25.0998752Z ----------------------------------------------------------------------
2022-05-18T04:29:25.3823324Z   test_all_reduce_full_group_max (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24343
2022-05-18T04:29:25.3845962Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24344
2022-05-18T04:29:25.3869281Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24345
2022-05-18T04:29:26.2229765Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:29:26.2235588Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:29:26.2236297Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:29:26.2237004Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:29:26.2237740Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:29:26.2331579Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:29:26.2344647Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:29:26.2345985Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:29:26.3343086Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:29:26.3458639Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:29:26.3559450Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-05-18T04:29:26.3560201Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:29:26.3560850Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:29:26.3561389Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:29:26.3561907Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:29:26.6923473Z ok (1.592s)
2022-05-18T04:29:26.6923781Z 
2022-05-18T04:29:26.6924277Z ----------------------------------------------------------------------
2022-05-18T04:29:26.6924547Z Ran 1 test in 1.592s
2022-05-18T04:29:26.6924662Z 
2022-05-18T04:29:26.6924726Z OK
2022-05-18T04:29:26.6924818Z 
2022-05-18T04:29:26.6924897Z Generating XML reports...
2022-05-18T04:29:26.6955460Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042925.xml
2022-05-18T04:29:27.6283176Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:29:27.6292899Z 
2022-05-18T04:29:27.6292998Z Running tests...
2022-05-18T04:29:27.6293583Z ----------------------------------------------------------------------
2022-05-18T04:29:27.9096393Z   test_all_reduce_full_group_min (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24408
2022-05-18T04:29:27.9117551Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24409
2022-05-18T04:29:27.9139984Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24410
2022-05-18T04:29:28.8245157Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:29:28.8345895Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:29:28.8346596Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:29:28.8347717Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:29:28.8348461Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:29:28.8348993Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:29:28.8454068Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:29:28.9363191Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:29:28.9363665Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:29:28.9566434Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:29:28.9667647Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:29:28.9668314Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-05-18T04:29:28.9668947Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:29:28.9669479Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:29:28.9669984Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:29:29.3197821Z ok (1.690s)
2022-05-18T04:29:29.3198063Z 
2022-05-18T04:29:29.3198527Z ----------------------------------------------------------------------
2022-05-18T04:29:29.3198909Z Ran 1 test in 1.690s
2022-05-18T04:29:29.3199086Z 
2022-05-18T04:29:29.3199181Z OK
2022-05-18T04:29:29.3199342Z 
2022-05-18T04:29:29.3199496Z Generating XML reports...
2022-05-18T04:29:29.3231284Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042927.xml
2022-05-18T04:29:30.2496240Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:29:30.2506356Z 
2022-05-18T04:29:30.2506496Z Running tests...
2022-05-18T04:29:30.2507002Z ----------------------------------------------------------------------
2022-05-18T04:29:30.5330221Z   test_all_reduce_full_group_product (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24473
2022-05-18T04:29:30.5352145Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24474
2022-05-18T04:29:30.5375120Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24475
2022-05-18T04:29:31.3359340Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:29:31.3459644Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:29:31.3460225Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:29:31.3461020Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:29:31.3461559Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:29:31.3462088Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:29:31.3567481Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:29:31.4473907Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:29:31.4474426Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:29:31.4780387Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:29:31.4882184Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-05-18T04:29:31.4882764Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:29:31.4883751Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:29:31.4884283Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:29:31.4884806Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:29:31.7427949Z ok (1.492s)
2022-05-18T04:29:31.7428096Z 
2022-05-18T04:29:31.7428462Z ----------------------------------------------------------------------
2022-05-18T04:29:31.7428980Z Ran 1 test in 1.492s
2022-05-18T04:29:31.7429097Z 
2022-05-18T04:29:31.7429160Z OK
2022-05-18T04:29:31.7429254Z 
2022-05-18T04:29:31.7429445Z Generating XML reports...
2022-05-18T04:29:31.7459422Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042930.xml
2022-05-18T04:29:32.6652200Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:29:32.6662661Z 
2022-05-18T04:29:32.6663062Z Running tests...
2022-05-18T04:29:32.6663497Z ----------------------------------------------------------------------
2022-05-18T04:29:32.9477870Z   test_all_reduce_full_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24538
2022-05-18T04:29:32.9499403Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24539
2022-05-18T04:29:32.9521897Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24540
2022-05-18T04:29:33.7309814Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:29:33.7411729Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:29:33.7412387Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:29:33.7412795Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:29:33.7413322Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:29:33.7413859Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:29:33.7421918Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:29:33.7423041Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:29:33.7424032Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:29:33.7631645Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:29:33.7632388Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:29:33.7633009Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-05-18T04:29:33.7633842Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:29:33.7634429Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:29:33.7634955Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:29:34.0573991Z ok (1.391s)
2022-05-18T04:29:34.0574232Z 
2022-05-18T04:29:34.0574782Z ----------------------------------------------------------------------
2022-05-18T04:29:34.0575066Z Ran 1 test in 1.391s
2022-05-18T04:29:34.0575182Z 
2022-05-18T04:29:34.0575244Z OK
2022-05-18T04:29:34.0575338Z 
2022-05-18T04:29:34.0575433Z Generating XML reports...
2022-05-18T04:29:34.0607236Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042932.xml
2022-05-18T04:29:34.9765415Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:29:34.9776293Z 
2022-05-18T04:29:34.9776771Z Running tests...
2022-05-18T04:29:34.9777198Z ----------------------------------------------------------------------
2022-05-18T04:29:35.2602164Z   test_all_reduce_group_max (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24603
2022-05-18T04:29:35.2623950Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24604
2022-05-18T04:29:35.2646908Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24605
2022-05-18T04:29:36.0854085Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:29:36.0954994Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:29:36.0955957Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:29:36.0956644Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:29:36.0957203Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:29:36.0957768Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:29:36.0964192Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:29:36.0965014Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:29:36.0966655Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:29:36.0967312Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:29:36.1172475Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:29:36.1172931Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-05-18T04:29:36.1173520Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:29:36.1174065Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:29:36.1271630Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:29:36.4698248Z ok (1.492s)
2022-05-18T04:29:36.4698498Z 
2022-05-18T04:29:36.4699032Z ----------------------------------------------------------------------
2022-05-18T04:29:36.4699376Z Ran 1 test in 1.492s
2022-05-18T04:29:36.4699497Z 
2022-05-18T04:29:36.4699562Z OK
2022-05-18T04:29:36.4701214Z 
2022-05-18T04:29:36.4701389Z Generating XML reports...
2022-05-18T04:29:36.4730829Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042934.xml
2022-05-18T04:29:37.4191372Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:29:37.4202176Z 
2022-05-18T04:29:37.4202277Z Running tests...
2022-05-18T04:29:37.4202880Z ----------------------------------------------------------------------
2022-05-18T04:29:37.7016063Z   test_all_reduce_group_min (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24664
2022-05-18T04:29:37.7038135Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24665
2022-05-18T04:29:37.7060808Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24666
2022-05-18T04:29:38.5474638Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:29:38.5575269Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:29:38.5575857Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:29:38.5576472Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:29:38.5577191Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:29:38.5578165Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:29:38.5586797Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:29:38.5587609Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:29:38.5588436Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:29:38.5588813Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:29:38.5795817Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-05-18T04:29:38.5796565Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:29:38.5797213Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:29:38.5797745Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:29:38.5893535Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:29:38.9112721Z ok (1.491s)
2022-05-18T04:29:38.9112970Z 
2022-05-18T04:29:38.9113486Z ----------------------------------------------------------------------
2022-05-18T04:29:38.9113854Z Ran 1 test in 1.491s
2022-05-18T04:29:38.9113972Z 
2022-05-18T04:29:38.9114034Z OK
2022-05-18T04:29:38.9114124Z 
2022-05-18T04:29:38.9114306Z Generating XML reports...
2022-05-18T04:29:38.9146245Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042937.xml
2022-05-18T04:29:39.8324289Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:29:39.8334009Z 
2022-05-18T04:29:39.8334130Z Running tests...
2022-05-18T04:29:39.8334593Z ----------------------------------------------------------------------
2022-05-18T04:29:40.1139165Z   test_all_reduce_group_product (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24725
2022-05-18T04:29:40.1161043Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24726
2022-05-18T04:29:40.1183496Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24727
2022-05-18T04:29:40.9198416Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:29:40.9299188Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:29:40.9299800Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:29:40.9300754Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:29:40.9301637Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:29:40.9302488Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:29:40.9407388Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:29:41.0313462Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:29:41.0314108Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:29:41.0315511Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:29:41.0518811Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:29:41.0519517Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-05-18T04:29:41.0520517Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:29:41.0521398Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:29:41.0522221Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:29:41.3237600Z ok (1.490s)
2022-05-18T04:29:41.3237814Z 
2022-05-18T04:29:41.3238335Z ----------------------------------------------------------------------
2022-05-18T04:29:41.3238774Z Ran 1 test in 1.490s
2022-05-18T04:29:41.3238984Z 
2022-05-18T04:29:41.3239101Z OK
2022-05-18T04:29:41.3239267Z 
2022-05-18T04:29:41.3239457Z Generating XML reports...
2022-05-18T04:29:41.3269153Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042939.xml
2022-05-18T04:29:42.2404019Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:29:42.2415057Z 
2022-05-18T04:29:42.2415382Z Running tests...
2022-05-18T04:29:42.2416039Z ----------------------------------------------------------------------
2022-05-18T04:29:42.5238158Z   test_all_reduce_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24786
2022-05-18T04:29:42.5260610Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24787
2022-05-18T04:29:42.5283174Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24788
2022-05-18T04:29:43.3446525Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:29:43.3546831Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:29:43.3547765Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:29:43.3549041Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:29:43.3550126Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:29:43.3550910Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:29:43.3558731Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:29:43.3559385Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:29:43.3560498Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:29:43.3561006Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:29:43.3765202Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-05-18T04:29:43.3765909Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:29:43.3766481Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:29:43.3767012Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:29:43.3865492Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:29:43.7335350Z ok (1.492s)
2022-05-18T04:29:43.7335561Z 
2022-05-18T04:29:43.7335883Z ----------------------------------------------------------------------
2022-05-18T04:29:43.7336326Z Ran 1 test in 1.492s
2022-05-18T04:29:43.7336441Z 
2022-05-18T04:29:43.7336503Z OK
2022-05-18T04:29:43.7336595Z 
2022-05-18T04:29:43.7336690Z Generating XML reports...
2022-05-18T04:29:43.7367430Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042942.xml
2022-05-18T04:29:44.6529643Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:29:44.6539455Z 
2022-05-18T04:29:44.6539549Z Running tests...
2022-05-18T04:29:44.6540278Z ----------------------------------------------------------------------
2022-05-18T04:29:44.9366980Z   test_all_reduce_max (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24847
2022-05-18T04:29:44.9389536Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24848
2022-05-18T04:29:44.9412708Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24849
2022-05-18T04:29:45.7588154Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:29:45.7690601Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:29:45.7691349Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:29:45.7691824Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:29:45.7692355Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:29:45.7692892Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:29:45.7699130Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:29:45.7700410Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:29:45.7701360Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:29:46.0464276Z ok (1.392s)
2022-05-18T04:29:46.0464539Z 
2022-05-18T04:29:46.0465066Z ----------------------------------------------------------------------
2022-05-18T04:29:46.0465312Z Ran 1 test in 1.392s
2022-05-18T04:29:46.0465427Z 
2022-05-18T04:29:46.0465489Z OK
2022-05-18T04:29:46.0465580Z 
2022-05-18T04:29:46.0465675Z Generating XML reports...
2022-05-18T04:29:46.0496172Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042944.xml
2022-05-18T04:29:46.9683934Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:29:46.9693794Z 
2022-05-18T04:29:46.9693908Z Running tests...
2022-05-18T04:29:46.9694764Z ----------------------------------------------------------------------
2022-05-18T04:29:47.2512830Z   test_all_reduce_min (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24903
2022-05-18T04:29:47.2535845Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24904
2022-05-18T04:29:47.2558528Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24905
2022-05-18T04:29:48.0684450Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:29:48.0684954Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:29:48.0685312Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:29:48.0685920Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:29:48.0686550Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:29:48.0687349Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:29:48.0790918Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:29:48.1696807Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:29:48.1697229Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:29:48.4611454Z ok (1.491s)
2022-05-18T04:29:48.4611731Z 
2022-05-18T04:29:48.4612220Z ----------------------------------------------------------------------
2022-05-18T04:29:48.4612481Z Ran 1 test in 1.492s
2022-05-18T04:29:48.4612598Z 
2022-05-18T04:29:48.4612661Z OK
2022-05-18T04:29:48.4612756Z 
2022-05-18T04:29:48.4612840Z Generating XML reports...
2022-05-18T04:29:48.4642659Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042946.xml
2022-05-18T04:29:49.3855052Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:29:49.3865267Z 
2022-05-18T04:29:49.3865344Z Running tests...
2022-05-18T04:29:49.3866301Z ----------------------------------------------------------------------
2022-05-18T04:29:49.6671196Z   test_all_reduce_multigpu (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24959
2022-05-18T04:29:49.6693046Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24960
2022-05-18T04:29:49.6716373Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24961
2022-05-18T04:29:50.4552975Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:29:50.4553603Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:29:50.4554003Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:29:50.4554684Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:29:50.4555213Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:29:50.4555730Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:29:50.4563147Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:29:50.4563609Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:29:50.4565324Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:29:50.6764838Z skip: CUDA is not available. (1.290s)
2022-05-18T04:29:50.6765136Z 
2022-05-18T04:29:50.6765601Z ----------------------------------------------------------------------
2022-05-18T04:29:50.6765970Z Ran 1 test in 1.290s
2022-05-18T04:29:50.6766086Z 
2022-05-18T04:29:50.6766172Z OK (skipped=1)
2022-05-18T04:29:50.6766287Z 
2022-05-18T04:29:50.6767560Z Generating XML reports...
2022-05-18T04:29:50.6796927Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042949.xml
2022-05-18T04:29:51.5976441Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:29:51.5986725Z 
2022-05-18T04:29:51.5986858Z Running tests...
2022-05-18T04:29:51.5987470Z ----------------------------------------------------------------------
2022-05-18T04:29:51.8783046Z   test_all_reduce_multigpu_complex (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25012
2022-05-18T04:29:51.8805137Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25013
2022-05-18T04:29:51.8828041Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25014
2022-05-18T04:29:52.7015312Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:29:52.7064554Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:29:52.7065010Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:29:52.7065609Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:29:52.7066142Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:29:52.7115920Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:29:52.7174280Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:29:52.7175022Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:29:52.8130359Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:29:52.9880838Z skip: CUDA is not available. (1.389s)
2022-05-18T04:29:52.9881184Z 
2022-05-18T04:29:52.9881664Z ----------------------------------------------------------------------
2022-05-18T04:29:52.9881918Z Ran 1 test in 1.389s
2022-05-18T04:29:52.9882021Z 
2022-05-18T04:29:52.9882096Z OK (skipped=1)
2022-05-18T04:29:52.9882205Z 
2022-05-18T04:29:52.9882294Z Generating XML reports...
2022-05-18T04:29:52.9914368Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042951.xml
2022-05-18T04:29:53.9089836Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:29:53.9099870Z 
2022-05-18T04:29:53.9099966Z Running tests...
2022-05-18T04:29:53.9100684Z ----------------------------------------------------------------------
2022-05-18T04:29:54.1904160Z   test_all_reduce_product (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25065
2022-05-18T04:29:54.1926060Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25066
2022-05-18T04:29:54.1948548Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25067
2022-05-18T04:29:55.0299243Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:29:55.0400552Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:29:55.0401116Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:29:55.0401720Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:29:55.0402394Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:29:55.0403225Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:29:55.0508035Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:29:55.1414509Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:29:55.1414883Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:29:55.4001229Z ok (1.490s)
2022-05-18T04:29:55.4001696Z 
2022-05-18T04:29:55.4002664Z ----------------------------------------------------------------------
2022-05-18T04:29:55.4003127Z Ran 1 test in 1.490s
2022-05-18T04:29:55.4003234Z 
2022-05-18T04:29:55.4003508Z OK
2022-05-18T04:29:55.4003601Z 
2022-05-18T04:29:55.4003702Z Generating XML reports...
2022-05-18T04:29:55.4033963Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042953.xml
2022-05-18T04:29:56.3261423Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:29:56.3272071Z 
2022-05-18T04:29:56.3272495Z Running tests...
2022-05-18T04:29:56.3272921Z ----------------------------------------------------------------------
2022-05-18T04:29:56.6093298Z   test_all_reduce_result_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25121
2022-05-18T04:29:56.6116147Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25122
2022-05-18T04:29:56.6138297Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25123
2022-05-18T04:29:57.4478940Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:29:57.4578505Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:29:57.4579101Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:29:57.4580052Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:29:57.4580877Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:29:57.4581777Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:29:57.4688003Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:29:57.4688700Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:29:57.5594690Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:29:57.7189229Z skip: CUDA is not available. (1.391s)
2022-05-18T04:29:57.7189521Z 
2022-05-18T04:29:57.7190073Z ----------------------------------------------------------------------
2022-05-18T04:29:57.7190405Z Ran 1 test in 1.392s
2022-05-18T04:29:57.7190519Z 
2022-05-18T04:29:57.7190593Z OK (skipped=1)
2022-05-18T04:29:57.7190702Z 
2022-05-18T04:29:57.7190787Z Generating XML reports...
2022-05-18T04:29:57.7221083Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042956.xml
2022-05-18T04:29:58.6426350Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:29:58.6436677Z 
2022-05-18T04:29:58.6436777Z Running tests...
2022-05-18T04:29:58.6437570Z ----------------------------------------------------------------------
2022-05-18T04:29:58.9252840Z   test_all_reduce_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25174
2022-05-18T04:29:58.9276281Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25175
2022-05-18T04:29:58.9298705Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25176
2022-05-18T04:29:59.7436262Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:29:59.7479960Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:29:59.7480410Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:29:59.7481070Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:29:59.7481601Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:29:59.7537439Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:29:59.7589051Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:29:59.7589696Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:29:59.8547929Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:30:00.1351598Z ok (1.491s)
2022-05-18T04:30:00.1351835Z 
2022-05-18T04:30:00.1352222Z ----------------------------------------------------------------------
2022-05-18T04:30:00.1352493Z Ran 1 test in 1.491s
2022-05-18T04:30:00.1352615Z 
2022-05-18T04:30:00.1352683Z OK
2022-05-18T04:30:00.1352775Z 
2022-05-18T04:30:00.1352870Z Generating XML reports...
2022-05-18T04:30:00.1385869Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042958.xml
2022-05-18T04:30:01.0557286Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:30:01.0567562Z 
2022-05-18T04:30:01.0567660Z Running tests...
2022-05-18T04:30:01.0568267Z ----------------------------------------------------------------------
2022-05-18T04:30:01.3405551Z   test_all_reduce_sum_async (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25230
2022-05-18T04:30:01.3428336Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25231
2022-05-18T04:30:01.3451598Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25232
2022-05-18T04:30:02.1449111Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:30:02.1449875Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:30:02.1450250Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:30:02.1450897Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:30:02.1451411Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:30:02.1451935Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:30:02.1460068Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:30:02.1460753Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:30:02.1461870Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:30:02.4503530Z ok (1.393s)
2022-05-18T04:30:02.4503722Z 
2022-05-18T04:30:02.4504066Z ----------------------------------------------------------------------
2022-05-18T04:30:02.4504348Z Ran 1 test in 1.393s
2022-05-18T04:30:02.4504463Z 
2022-05-18T04:30:02.4504527Z OK
2022-05-18T04:30:02.4504622Z 
2022-05-18T04:30:02.4504712Z Generating XML reports...
2022-05-18T04:30:02.4535171Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043001.xml
2022-05-18T04:30:03.4078198Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:30:03.4088156Z 
2022-05-18T04:30:03.4088396Z Running tests...
2022-05-18T04:30:03.4088734Z ----------------------------------------------------------------------
2022-05-18T04:30:03.6937271Z   test_all_reduce_sum_complex (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25286
2022-05-18T04:30:03.6959775Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25287
2022-05-18T04:30:03.6981764Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25288
2022-05-18T04:30:04.4943727Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:30:04.5045478Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:30:04.5045957Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:30:04.5046574Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:30:04.5047134Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:30:04.5047975Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:30:04.5056350Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:30:04.5056822Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:30:04.5057992Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:30:04.8036304Z ok (1.394s)
2022-05-18T04:30:04.8036557Z 
2022-05-18T04:30:04.8037097Z ----------------------------------------------------------------------
2022-05-18T04:30:04.8037348Z Ran 1 test in 1.395s
2022-05-18T04:30:04.8037465Z 
2022-05-18T04:30:04.8037513Z OK
2022-05-18T04:30:04.8037605Z 
2022-05-18T04:30:04.8037698Z Generating XML reports...
2022-05-18T04:30:04.8068457Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043003.xml
2022-05-18T04:30:05.7401326Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:30:05.7410723Z 
2022-05-18T04:30:05.7410834Z Running tests...
2022-05-18T04:30:05.7411441Z ----------------------------------------------------------------------
2022-05-18T04:30:06.0253464Z   test_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25342
2022-05-18T04:30:06.0276055Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25343
2022-05-18T04:30:06.0298889Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25344
2022-05-18T04:30:06.8550526Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:30:06.8578594Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:30:06.8579155Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:30:06.8579948Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:30:06.8580543Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:30:06.8651565Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:30:06.8688763Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:30:06.8689329Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:30:06.9664371Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:30:07.1349379Z skip: CUDA is not available. (1.394s)
2022-05-18T04:30:07.1349673Z 
2022-05-18T04:30:07.1350218Z ----------------------------------------------------------------------
2022-05-18T04:30:07.1350519Z Ran 1 test in 1.394s
2022-05-18T04:30:07.1350633Z 
2022-05-18T04:30:07.1350708Z OK (skipped=1)
2022-05-18T04:30:07.1350817Z 
2022-05-18T04:30:07.1350904Z Generating XML reports...
2022-05-18T04:30:07.1381490Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043005.xml
2022-05-18T04:30:08.0825210Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:30:08.0835061Z 
2022-05-18T04:30:08.0835166Z Running tests...
2022-05-18T04:30:08.0835599Z ----------------------------------------------------------------------
2022-05-18T04:30:08.3662757Z   test_all_reduce_sum_cuda_async (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25395
2022-05-18T04:30:08.3684961Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25396
2022-05-18T04:30:08.3708233Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25397
2022-05-18T04:30:09.2011447Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:30:09.2112871Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:30:09.2113413Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:30:09.2114118Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:30:09.2114703Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:30:09.2115553Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:30:09.2124094Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:30:09.2126264Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:30:09.2126843Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:30:09.3757732Z skip: CUDA is not available. (1.292s)
2022-05-18T04:30:09.3757908Z 
2022-05-18T04:30:09.3758245Z ----------------------------------------------------------------------
2022-05-18T04:30:09.3758539Z Ran 1 test in 1.292s
2022-05-18T04:30:09.3758655Z 
2022-05-18T04:30:09.3758728Z OK (skipped=1)
2022-05-18T04:30:09.3758862Z 
2022-05-18T04:30:09.3758972Z Generating XML reports...
2022-05-18T04:30:09.3788798Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043008.xml
2022-05-18T04:30:10.3131769Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:30:10.3141640Z 
2022-05-18T04:30:10.3141781Z Running tests...
2022-05-18T04:30:10.3142210Z ----------------------------------------------------------------------
2022-05-18T04:30:10.5971994Z   test_all_reduce_sum_cuda_complex (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25448
2022-05-18T04:30:10.5995820Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25449
2022-05-18T04:30:10.6019084Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25450
2022-05-18T04:30:11.4243893Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:30:11.4339045Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:30:11.4339485Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:30:11.4340100Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:30:11.4340615Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:30:11.4345550Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:30:11.4448766Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:30:11.4449365Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:30:11.5358492Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:30:11.7069989Z skip: CUDA is not available. (1.393s)
2022-05-18T04:30:11.7070266Z 
2022-05-18T04:30:11.7070689Z ----------------------------------------------------------------------
2022-05-18T04:30:11.7070946Z Ran 1 test in 1.393s
2022-05-18T04:30:11.7071048Z 
2022-05-18T04:30:11.7071128Z OK (skipped=1)
2022-05-18T04:30:11.7071237Z 
2022-05-18T04:30:11.7071325Z Generating XML reports...
2022-05-18T04:30:11.7102760Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043010.xml
2022-05-18T04:30:12.6530696Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:30:12.6540533Z 
2022-05-18T04:30:12.6541082Z Running tests...
2022-05-18T04:30:12.6541555Z ----------------------------------------------------------------------
2022-05-18T04:30:12.6555857Z   test_all_to_all (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.001s)
2022-05-18T04:30:12.6556185Z 
2022-05-18T04:30:12.6556522Z ----------------------------------------------------------------------
2022-05-18T04:30:12.6556948Z Ran 1 test in 0.002s
2022-05-18T04:30:12.6557155Z 
2022-05-18T04:30:12.6557267Z OK (skipped=1)
2022-05-18T04:30:12.6557465Z 
2022-05-18T04:30:12.6558574Z Generating XML reports...
2022-05-18T04:30:12.6590093Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043012.xml
2022-05-18T04:30:13.5000836Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:30:13.5010991Z 
2022-05-18T04:30:13.5011280Z Running tests...
2022-05-18T04:30:13.5011926Z ----------------------------------------------------------------------
2022-05-18T04:30:13.5026762Z   test_all_to_all_complex (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.001s)
2022-05-18T04:30:13.5027185Z 
2022-05-18T04:30:13.5027606Z ----------------------------------------------------------------------
2022-05-18T04:30:13.5027847Z Ran 1 test in 0.002s
2022-05-18T04:30:13.5027963Z 
2022-05-18T04:30:13.5028038Z OK (skipped=1)
2022-05-18T04:30:13.5028152Z 
2022-05-18T04:30:13.5028238Z Generating XML reports...
2022-05-18T04:30:13.5059516Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043013.xml
2022-05-18T04:30:14.3508550Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:30:14.3518303Z 
2022-05-18T04:30:14.3518403Z Running tests...
2022-05-18T04:30:14.3519576Z ----------------------------------------------------------------------
2022-05-18T04:30:14.3535364Z   test_all_to_all_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only NCCL supports CUDA all_to_all (0.002s)
2022-05-18T04:30:14.3535805Z 
2022-05-18T04:30:14.3536376Z ----------------------------------------------------------------------
2022-05-18T04:30:14.3536912Z Ran 1 test in 0.002s
2022-05-18T04:30:14.3537029Z 
2022-05-18T04:30:14.3537104Z OK (skipped=1)
2022-05-18T04:30:14.3537199Z 
2022-05-18T04:30:14.3537286Z Generating XML reports...
2022-05-18T04:30:14.3568874Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043014.xml
2022-05-18T04:30:15.2012869Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:30:15.2023951Z 
2022-05-18T04:30:15.2024343Z Running tests...
2022-05-18T04:30:15.2024779Z ----------------------------------------------------------------------
2022-05-18T04:30:15.2041108Z   test_all_to_all_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only NCCL supports CUDA all_to_all (0.002s)
2022-05-18T04:30:15.2041505Z 
2022-05-18T04:30:15.2042093Z ----------------------------------------------------------------------
2022-05-18T04:30:15.2042505Z Ran 1 test in 0.002s
2022-05-18T04:30:15.2042702Z 
2022-05-18T04:30:15.2042807Z OK (skipped=1)
2022-05-18T04:30:15.2042994Z 
2022-05-18T04:30:15.2043136Z Generating XML reports...
2022-05-18T04:30:15.2074337Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043015.xml
2022-05-18T04:30:16.0653481Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:30:16.0663716Z 
2022-05-18T04:30:16.0663847Z Running tests...
2022-05-18T04:30:16.0664328Z ----------------------------------------------------------------------
2022-05-18T04:30:16.0680144Z   test_all_to_all_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.001s)
2022-05-18T04:30:16.0680686Z 
2022-05-18T04:30:16.0680957Z ----------------------------------------------------------------------
2022-05-18T04:30:16.0681215Z Ran 1 test in 0.002s
2022-05-18T04:30:16.0681339Z 
2022-05-18T04:30:16.0681416Z OK (skipped=1)
2022-05-18T04:30:16.0681522Z 
2022-05-18T04:30:16.0681609Z Generating XML reports...
2022-05-18T04:30:16.0715747Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043016.xml
2022-05-18T04:30:16.9342324Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:30:16.9352235Z 
2022-05-18T04:30:16.9352332Z Running tests...
2022-05-18T04:30:16.9353002Z ----------------------------------------------------------------------
2022-05-18T04:30:16.9368411Z   test_all_to_all_full_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only NCCL supports CUDA all_to_all (0.001s)
2022-05-18T04:30:16.9368868Z 
2022-05-18T04:30:16.9369308Z ----------------------------------------------------------------------
2022-05-18T04:30:16.9369808Z Ran 1 test in 0.002s
2022-05-18T04:30:16.9369926Z 
2022-05-18T04:30:16.9369986Z OK (skipped=1)
2022-05-18T04:30:16.9370101Z 
2022-05-18T04:30:16.9370189Z Generating XML reports...
2022-05-18T04:30:16.9406078Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043016.xml
2022-05-18T04:30:17.7974727Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:30:17.7985602Z 
2022-05-18T04:30:17.7985828Z Running tests...
2022-05-18T04:30:17.7986436Z ----------------------------------------------------------------------
2022-05-18T04:30:17.8001907Z   test_all_to_all_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.001s)
2022-05-18T04:30:17.8002296Z 
2022-05-18T04:30:17.8002701Z ----------------------------------------------------------------------
2022-05-18T04:30:17.8003075Z Ran 1 test in 0.002s
2022-05-18T04:30:17.8003188Z 
2022-05-18T04:30:17.8003261Z OK (skipped=1)
2022-05-18T04:30:17.8003368Z 
2022-05-18T04:30:17.8003450Z Generating XML reports...
2022-05-18T04:30:17.8035192Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043017.xml
2022-05-18T04:30:18.6466080Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:30:18.6475514Z 
2022-05-18T04:30:18.6475654Z Running tests...
2022-05-18T04:30:18.6476038Z ----------------------------------------------------------------------
2022-05-18T04:30:18.6492037Z   test_all_to_all_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.001s)
2022-05-18T04:30:18.6492379Z 
2022-05-18T04:30:18.6492707Z ----------------------------------------------------------------------
2022-05-18T04:30:18.6492960Z Ran 1 test in 0.002s
2022-05-18T04:30:18.6493072Z 
2022-05-18T04:30:18.6493376Z OK (skipped=1)
2022-05-18T04:30:18.6493503Z 
2022-05-18T04:30:18.6493592Z Generating XML reports...
2022-05-18T04:30:18.6523954Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043018.xml
2022-05-18T04:30:19.4957584Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:30:19.4967570Z 
2022-05-18T04:30:19.4967681Z Running tests...
2022-05-18T04:30:19.4968232Z ----------------------------------------------------------------------
2022-05-18T04:30:19.4984951Z   test_all_to_all_single_equal_split (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.001s)
2022-05-18T04:30:19.4985409Z 
2022-05-18T04:30:19.4985690Z ----------------------------------------------------------------------
2022-05-18T04:30:19.4985954Z Ran 1 test in 0.002s
2022-05-18T04:30:19.4986069Z 
2022-05-18T04:30:19.4986142Z OK (skipped=1)
2022-05-18T04:30:19.4986249Z 
2022-05-18T04:30:19.4986352Z Generating XML reports...
2022-05-18T04:30:19.5017127Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043019.xml
2022-05-18T04:30:20.3584399Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:30:20.3595350Z 
2022-05-18T04:30:20.3595659Z Running tests...
2022-05-18T04:30:20.3596257Z ----------------------------------------------------------------------
2022-05-18T04:30:20.3612233Z   test_all_to_all_single_equal_split_complex (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-05-18T04:30:20.3612714Z 
2022-05-18T04:30:20.3612973Z ----------------------------------------------------------------------
2022-05-18T04:30:20.3613241Z Ran 1 test in 0.002s
2022-05-18T04:30:20.3613355Z 
2022-05-18T04:30:20.3613416Z OK (skipped=1)
2022-05-18T04:30:20.3613527Z 
2022-05-18T04:30:20.3613612Z Generating XML reports...
2022-05-18T04:30:20.3646878Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043020.xml
2022-05-18T04:30:21.2064005Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:30:21.2074302Z 
2022-05-18T04:30:21.2074755Z Running tests...
2022-05-18T04:30:21.2075150Z ----------------------------------------------------------------------
2022-05-18T04:30:21.2091365Z   test_all_to_all_single_equal_split_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-05-18T04:30:21.2091756Z 
2022-05-18T04:30:21.2092028Z ----------------------------------------------------------------------
2022-05-18T04:30:21.2092279Z Ran 1 test in 0.002s
2022-05-18T04:30:21.2092395Z 
2022-05-18T04:30:21.2092456Z OK (skipped=1)
2022-05-18T04:30:21.2092563Z 
2022-05-18T04:30:21.2092653Z Generating XML reports...
2022-05-18T04:30:21.2123693Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043021.xml
2022-05-18T04:30:22.0410516Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:30:22.0420760Z 
2022-05-18T04:30:22.0421095Z Running tests...
2022-05-18T04:30:22.0421726Z ----------------------------------------------------------------------
2022-05-18T04:30:22.0438273Z   test_all_to_all_single_equal_split_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-05-18T04:30:22.0438733Z 
2022-05-18T04:30:22.0439148Z ----------------------------------------------------------------------
2022-05-18T04:30:22.0439461Z Ran 1 test in 0.002s
2022-05-18T04:30:22.0439564Z 
2022-05-18T04:30:22.0439639Z OK (skipped=1)
2022-05-18T04:30:22.0439745Z 
2022-05-18T04:30:22.0439829Z Generating XML reports...
2022-05-18T04:30:22.0470627Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043022.xml
2022-05-18T04:30:22.8787793Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:30:22.8797953Z 
2022-05-18T04:30:22.8798056Z Running tests...
2022-05-18T04:30:22.8798719Z ----------------------------------------------------------------------
2022-05-18T04:30:22.8814858Z   test_all_to_all_single_equal_split_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.001s)
2022-05-18T04:30:22.8815146Z 
2022-05-18T04:30:22.8815416Z ----------------------------------------------------------------------
2022-05-18T04:30:22.8815649Z Ran 1 test in 0.002s
2022-05-18T04:30:22.8815764Z 
2022-05-18T04:30:22.8815837Z OK (skipped=1)
2022-05-18T04:30:22.8815947Z 
2022-05-18T04:30:22.8816034Z Generating XML reports...
2022-05-18T04:30:22.8855816Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043022.xml
2022-05-18T04:30:23.7139010Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:30:23.7149032Z 
2022-05-18T04:30:23.7149130Z Running tests...
2022-05-18T04:30:23.7150246Z ----------------------------------------------------------------------
2022-05-18T04:30:23.7165689Z   test_all_to_all_single_equal_split_full_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-05-18T04:30:23.7166174Z 
2022-05-18T04:30:23.7166516Z ----------------------------------------------------------------------
2022-05-18T04:30:23.7166767Z Ran 1 test in 0.002s
2022-05-18T04:30:23.7166883Z 
2022-05-18T04:30:23.7166942Z OK (skipped=1)
2022-05-18T04:30:23.7167051Z 
2022-05-18T04:30:23.7167138Z Generating XML reports...
2022-05-18T04:30:23.7197757Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043023.xml
2022-05-18T04:30:24.5495996Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:30:24.5506225Z 
2022-05-18T04:30:24.5506487Z Running tests...
2022-05-18T04:30:24.5507187Z ----------------------------------------------------------------------
2022-05-18T04:30:24.5522265Z   test_all_to_all_single_equal_split_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.001s)
2022-05-18T04:30:24.5522723Z 
2022-05-18T04:30:24.5522991Z ----------------------------------------------------------------------
2022-05-18T04:30:24.5523225Z Ran 1 test in 0.002s
2022-05-18T04:30:24.5523340Z 
2022-05-18T04:30:24.5523413Z OK (skipped=1)
2022-05-18T04:30:24.5523521Z 
2022-05-18T04:30:24.5523606Z Generating XML reports...
2022-05-18T04:30:24.5553859Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043024.xml
2022-05-18T04:30:25.3845037Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:30:25.3855280Z 
2022-05-18T04:30:25.3855751Z Running tests...
2022-05-18T04:30:25.3856162Z ----------------------------------------------------------------------
2022-05-18T04:30:25.3872167Z   test_all_to_all_single_equal_split_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-05-18T04:30:25.3872590Z 
2022-05-18T04:30:25.3873011Z ----------------------------------------------------------------------
2022-05-18T04:30:25.3873345Z Ran 1 test in 0.002s
2022-05-18T04:30:25.3883903Z 
2022-05-18T04:30:25.3883996Z OK (skipped=1)
2022-05-18T04:30:25.3884114Z 
2022-05-18T04:30:25.3884193Z Generating XML reports...
2022-05-18T04:30:25.3905061Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043025.xml
2022-05-18T04:30:26.2202518Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:30:26.2212642Z 
2022-05-18T04:30:26.2212751Z Running tests...
2022-05-18T04:30:26.2213375Z ----------------------------------------------------------------------
2022-05-18T04:30:26.2229021Z   test_all_to_all_single_unequal_split (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.001s)
2022-05-18T04:30:26.2229425Z 
2022-05-18T04:30:26.2229696Z ----------------------------------------------------------------------
2022-05-18T04:30:26.2229947Z Ran 1 test in 0.002s
2022-05-18T04:30:26.2230062Z 
2022-05-18T04:30:26.2230140Z OK (skipped=1)
2022-05-18T04:30:26.2230235Z 
2022-05-18T04:30:26.2230323Z Generating XML reports...
2022-05-18T04:30:26.2261903Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043026.xml
2022-05-18T04:30:27.0569390Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:30:27.0579180Z 
2022-05-18T04:30:27.0579270Z Running tests...
2022-05-18T04:30:27.0580335Z ----------------------------------------------------------------------
2022-05-18T04:30:27.0594720Z   test_all_to_all_single_unequal_split_complex (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.001s)
2022-05-18T04:30:27.0595461Z 
2022-05-18T04:30:27.0595737Z ----------------------------------------------------------------------
2022-05-18T04:30:27.0596192Z Ran 1 test in 0.002s
2022-05-18T04:30:27.0596400Z 
2022-05-18T04:30:27.0596507Z OK (skipped=1)
2022-05-18T04:30:27.0596625Z 
2022-05-18T04:30:27.0596710Z Generating XML reports...
2022-05-18T04:30:27.0627005Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043027.xml
2022-05-18T04:30:27.8938062Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:30:27.8948996Z 
2022-05-18T04:30:27.8949252Z Running tests...
2022-05-18T04:30:27.8949882Z ----------------------------------------------------------------------
2022-05-18T04:30:27.8965593Z   test_all_to_all_single_unequal_split_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-05-18T04:30:27.8966011Z 
2022-05-18T04:30:27.8966387Z ----------------------------------------------------------------------
2022-05-18T04:30:27.8966674Z Ran 1 test in 0.002s
2022-05-18T04:30:27.8966789Z 
2022-05-18T04:30:27.8966864Z OK (skipped=1)
2022-05-18T04:30:27.8966976Z 
2022-05-18T04:30:27.8967060Z Generating XML reports...
2022-05-18T04:30:27.9009998Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043027.xml
2022-05-18T04:30:28.7325383Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:30:28.7335071Z 
2022-05-18T04:30:28.7335187Z Running tests...
2022-05-18T04:30:28.7335771Z ----------------------------------------------------------------------
2022-05-18T04:30:28.7351919Z   test_all_to_all_single_unequal_split_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-05-18T04:30:28.7352341Z 
2022-05-18T04:30:28.7352676Z ----------------------------------------------------------------------
2022-05-18T04:30:28.7352932Z Ran 1 test in 0.002s
2022-05-18T04:30:28.7353047Z 
2022-05-18T04:30:28.7353126Z OK (skipped=1)
2022-05-18T04:30:28.7353222Z 
2022-05-18T04:30:28.7353308Z Generating XML reports...
2022-05-18T04:30:28.7385348Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043028.xml
2022-05-18T04:30:29.5700508Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:30:29.5710642Z 
2022-05-18T04:30:29.5710782Z Running tests...
2022-05-18T04:30:29.5711479Z ----------------------------------------------------------------------
2022-05-18T04:30:29.5726218Z   test_all_to_all_single_unequal_split_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.001s)
2022-05-18T04:30:29.5726958Z 
2022-05-18T04:30:29.5727386Z ----------------------------------------------------------------------
2022-05-18T04:30:29.5727828Z Ran 1 test in 0.002s
2022-05-18T04:30:29.5728033Z 
2022-05-18T04:30:29.5728277Z OK (skipped=1)
2022-05-18T04:30:29.5728472Z 
2022-05-18T04:30:29.5728626Z Generating XML reports...
2022-05-18T04:30:29.5758575Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043029.xml
2022-05-18T04:30:30.4041418Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:30:30.4051195Z 
2022-05-18T04:30:30.4051349Z Running tests...
2022-05-18T04:30:30.4051780Z ----------------------------------------------------------------------
2022-05-18T04:30:30.4069020Z   test_all_to_all_single_unequal_split_full_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-05-18T04:30:30.4069513Z 
2022-05-18T04:30:30.4069850Z ----------------------------------------------------------------------
2022-05-18T04:30:30.4070101Z Ran 1 test in 0.002s
2022-05-18T04:30:30.4070216Z 
2022-05-18T04:30:30.4070275Z OK (skipped=1)
2022-05-18T04:30:30.4070393Z 
2022-05-18T04:30:30.4070481Z Generating XML reports...
2022-05-18T04:30:30.4100380Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043030.xml
2022-05-18T04:30:31.2414558Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:30:31.2424767Z 
2022-05-18T04:30:31.2424858Z Running tests...
2022-05-18T04:30:31.2425430Z ----------------------------------------------------------------------
2022-05-18T04:30:31.2440240Z   test_all_to_all_single_unequal_split_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.001s)
2022-05-18T04:30:31.2440550Z 
2022-05-18T04:30:31.2440784Z ----------------------------------------------------------------------
2022-05-18T04:30:31.2441078Z Ran 1 test in 0.002s
2022-05-18T04:30:31.2441208Z 
2022-05-18T04:30:31.2441283Z OK (skipped=1)
2022-05-18T04:30:31.2441392Z 
2022-05-18T04:30:31.2441476Z Generating XML reports...
2022-05-18T04:30:31.2473363Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043031.xml
2022-05-18T04:30:32.0830227Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:30:32.0840246Z 
2022-05-18T04:30:32.0840350Z Running tests...
2022-05-18T04:30:32.0840750Z ----------------------------------------------------------------------
2022-05-18T04:30:32.0856950Z   test_all_to_all_single_unequal_split_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-05-18T04:30:32.0857413Z 
2022-05-18T04:30:32.0857842Z ----------------------------------------------------------------------
2022-05-18T04:30:32.0858282Z Ran 1 test in 0.002s
2022-05-18T04:30:32.0858415Z 
2022-05-18T04:30:32.0858486Z OK (skipped=1)
2022-05-18T04:30:32.0858593Z 
2022-05-18T04:30:32.0858679Z Generating XML reports...
2022-05-18T04:30:32.0889374Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043032.xml
2022-05-18T04:30:32.9155289Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:30:32.9165044Z 
2022-05-18T04:30:32.9165465Z Running tests...
2022-05-18T04:30:32.9165890Z ----------------------------------------------------------------------
2022-05-18T04:30:33.1978058Z   test_average_parameters (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25741
2022-05-18T04:30:33.2000273Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25742
2022-05-18T04:30:33.2023088Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25743
2022-05-18T04:30:33.9895151Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:30:33.9896071Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:30:33.9896666Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:30:33.9897705Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:30:33.9898281Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:30:33.9898796Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:30:34.0002172Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:30:34.0909822Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:30:34.0910387Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:30:34.3074814Z skip: Need at least 2 CUDA devices (1.391s)
2022-05-18T04:30:34.3075048Z 
2022-05-18T04:30:34.3075361Z ----------------------------------------------------------------------
2022-05-18T04:30:34.3075608Z Ran 1 test in 1.391s
2022-05-18T04:30:34.3075775Z 
2022-05-18T04:30:34.3075840Z OK (skipped=1)
2022-05-18T04:30:34.3075950Z 
2022-05-18T04:30:34.3076034Z Generating XML reports...
2022-05-18T04:30:34.3115723Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043032.xml
2022-05-18T04:30:35.2320051Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:30:35.2330765Z 
2022-05-18T04:30:35.2331065Z Running tests...
2022-05-18T04:30:35.2331709Z ----------------------------------------------------------------------
2022-05-18T04:30:35.2347410Z   test_backend_full_group (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'nccl'} (0.002s)
2022-05-18T04:30:35.2347681Z 
2022-05-18T04:30:35.2348055Z ----------------------------------------------------------------------
2022-05-18T04:30:35.2348503Z Ran 1 test in 0.002s
2022-05-18T04:30:35.2348690Z 
2022-05-18T04:30:35.2348771Z OK (skipped=1)
2022-05-18T04:30:35.2348879Z 
2022-05-18T04:30:35.2348951Z Generating XML reports...
2022-05-18T04:30:35.2379989Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043035.xml
2022-05-18T04:30:36.0671049Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:30:36.0681342Z 
2022-05-18T04:30:36.0681463Z Running tests...
2022-05-18T04:30:36.0682048Z ----------------------------------------------------------------------
2022-05-18T04:30:36.0697673Z   test_backend_group (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'nccl'} (0.001s)
2022-05-18T04:30:36.0698105Z 
2022-05-18T04:30:36.0698359Z ----------------------------------------------------------------------
2022-05-18T04:30:36.0698667Z Ran 1 test in 0.002s
2022-05-18T04:30:36.0698794Z 
2022-05-18T04:30:36.0698856Z OK (skipped=1)
2022-05-18T04:30:36.0699025Z 
2022-05-18T04:30:36.0699122Z Generating XML reports...
2022-05-18T04:30:36.0730489Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043036.xml
2022-05-18T04:30:36.8992960Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:30:36.9003926Z 
2022-05-18T04:30:36.9004239Z Running tests...
2022-05-18T04:30:36.9004875Z ----------------------------------------------------------------------
2022-05-18T04:30:37.1808891Z   test_barrier (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25814
2022-05-18T04:30:37.1831572Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25815
2022-05-18T04:30:37.1854765Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25816
2022-05-18T04:30:38.0061086Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:30:38.0162205Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:30:38.0162875Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:30:38.0163477Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:30:38.0164012Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:30:38.0164536Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:30:38.0171785Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:30:38.0173819Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:30:38.0174427Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:30:39.4921860Z ok (2.592s)
2022-05-18T04:30:39.4922100Z 
2022-05-18T04:30:39.4922540Z ----------------------------------------------------------------------
2022-05-18T04:30:39.4922938Z Ran 1 test in 2.592s
2022-05-18T04:30:39.4923130Z 
2022-05-18T04:30:39.4924457Z OK
2022-05-18T04:30:39.4924597Z 
2022-05-18T04:30:39.4924738Z Generating XML reports...
2022-05-18T04:30:39.4962119Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043036.xml
2022-05-18T04:30:40.4393820Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:30:40.4404455Z 
2022-05-18T04:30:40.4404842Z Running tests...
2022-05-18T04:30:40.4405272Z ----------------------------------------------------------------------
2022-05-18T04:30:40.7261088Z   test_barrier_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25867
2022-05-18T04:30:40.7283754Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25868
2022-05-18T04:30:40.7306308Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25869
2022-05-18T04:30:41.5340224Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:30:41.5441211Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:30:41.5441887Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:30:41.5442589Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:30:41.5443132Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:30:41.5443669Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:30:41.5453116Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:30:41.5453640Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:30:41.5454635Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:30:41.7354832Z skip: CUDA is not available. (1.295s)
2022-05-18T04:30:41.7355110Z 
2022-05-18T04:30:41.7355649Z ----------------------------------------------------------------------
2022-05-18T04:30:41.7355952Z Ran 1 test in 1.295s
2022-05-18T04:30:41.7356282Z 
2022-05-18T04:30:41.7356356Z OK (skipped=1)
2022-05-18T04:30:41.7356465Z 
2022-05-18T04:30:41.7356539Z Generating XML reports...
2022-05-18T04:30:41.7386531Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043040.xml
2022-05-18T04:30:42.6637842Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:30:42.6647701Z 
2022-05-18T04:30:42.6647798Z Running tests...
2022-05-18T04:30:42.6648290Z ----------------------------------------------------------------------
2022-05-18T04:30:42.9464790Z   test_barrier_full_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25920
2022-05-18T04:30:42.9487692Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25921
2022-05-18T04:30:42.9510208Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25922
2022-05-18T04:30:43.7434719Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:30:43.7435227Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:30:43.7435596Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:30:43.7436256Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:30:43.7437174Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:30:43.7437803Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:30:43.7444254Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:30:43.7444814Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:30:43.8446887Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:30:43.8659145Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:30:43.8659898Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:30:43.8660325Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-05-18T04:30:43.8661000Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:30:43.8661561Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:30:43.8662088Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:30:45.3578443Z ok (2.693s)
2022-05-18T04:30:45.3578714Z 
2022-05-18T04:30:45.3579246Z ----------------------------------------------------------------------
2022-05-18T04:30:45.3579584Z Ran 1 test in 2.693s
2022-05-18T04:30:45.3579718Z 
2022-05-18T04:30:45.3579780Z OK
2022-05-18T04:30:45.3579859Z 
2022-05-18T04:30:45.3579962Z Generating XML reports...
2022-05-18T04:30:45.3610666Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043042.xml
2022-05-18T04:30:46.3059881Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:30:46.3071091Z 
2022-05-18T04:30:46.3071410Z Running tests...
2022-05-18T04:30:46.3072052Z ----------------------------------------------------------------------
2022-05-18T04:30:46.5903548Z   test_barrier_full_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25982
2022-05-18T04:30:46.5927251Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25983
2022-05-18T04:30:46.5951009Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25984
2022-05-18T04:30:47.3887058Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:30:47.3987562Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:30:47.3987994Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:30:47.3988623Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:30:47.3989156Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:30:47.3989665Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:30:47.4095425Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:30:47.5002138Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:30:47.5002728Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:30:47.7002669Z skip: CUDA is not available. (1.393s)
2022-05-18T04:30:47.7002949Z 
2022-05-18T04:30:47.7003457Z ----------------------------------------------------------------------
2022-05-18T04:30:47.7003864Z Ran 1 test in 1.393s
2022-05-18T04:30:47.7003979Z 
2022-05-18T04:30:47.7004053Z OK (skipped=1)
2022-05-18T04:30:47.7004165Z 
2022-05-18T04:30:47.7004254Z Generating XML reports...
2022-05-18T04:30:47.7034500Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043046.xml
2022-05-18T04:30:48.6256789Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:30:48.6267081Z 
2022-05-18T04:30:48.6267210Z Running tests...
2022-05-18T04:30:48.6267815Z ----------------------------------------------------------------------
2022-05-18T04:30:48.9062795Z   test_barrier_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26035
2022-05-18T04:30:48.9085135Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26036
2022-05-18T04:30:48.9107908Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26037
2022-05-18T04:30:49.7016932Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:30:49.7017519Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:30:49.7017899Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:30:49.7018519Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:30:49.7019073Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:30:49.7019580Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:30:49.7123549Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:30:49.8030395Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:30:49.8030885Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:30:49.8031446Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:30:49.8235449Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-05-18T04:30:49.8236398Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:30:49.8237081Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:30:49.8237617Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:30:49.8238136Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:30:50.9171782Z ok (2.290s)
2022-05-18T04:30:50.9171950Z 
2022-05-18T04:30:50.9172398Z ----------------------------------------------------------------------
2022-05-18T04:30:50.9172861Z Ran 1 test in 2.290s
2022-05-18T04:30:50.9173038Z 
2022-05-18T04:30:50.9173108Z OK
2022-05-18T04:30:50.9173200Z 
2022-05-18T04:30:50.9173297Z Generating XML reports...
2022-05-18T04:30:50.9205218Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043048.xml
2022-05-18T04:30:51.8601417Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:30:51.8611427Z 
2022-05-18T04:30:51.8611516Z Running tests...
2022-05-18T04:30:51.8611987Z ----------------------------------------------------------------------
2022-05-18T04:30:52.1419142Z   test_barrier_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26094
2022-05-18T04:30:52.1441307Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26095
2022-05-18T04:30:52.1464531Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26096
2022-05-18T04:30:52.9866153Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:30:52.9866582Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:30:52.9866969Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:30:52.9867837Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:30:52.9868731Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:30:52.9869569Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:30:52.9972526Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:30:53.0880615Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:30:53.0881211Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:30:53.2517139Z skip: CUDA is not available. (1.390s)
2022-05-18T04:30:53.2517434Z 
2022-05-18T04:30:53.2517908Z ----------------------------------------------------------------------
2022-05-18T04:30:53.2518377Z Ran 1 test in 1.391s
2022-05-18T04:30:53.2518585Z 
2022-05-18T04:30:53.2518720Z OK (skipped=1)
2022-05-18T04:30:53.2518922Z 
2022-05-18T04:30:53.2520419Z Generating XML reports...
2022-05-18T04:30:53.2549966Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043051.xml
2022-05-18T04:30:54.1795323Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:30:54.1805194Z 
2022-05-18T04:30:54.1805328Z Running tests...
2022-05-18T04:30:54.1805745Z ----------------------------------------------------------------------
2022-05-18T04:30:54.4620400Z   test_barrier_timeout_full_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26147
2022-05-18T04:30:54.4643661Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26148
2022-05-18T04:30:54.4666065Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26149
2022-05-18T04:30:55.2600503Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:30:55.2702110Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:30:55.2702616Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:30:55.2703421Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:30:55.2703995Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:30:55.2704518Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:30:55.2712894Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:30:55.2714052Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:30:55.2714614Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:30:55.2919241Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:30:55.2921632Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-05-18T04:30:55.2922110Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:30:55.2922772Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:30:55.2923324Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:30:55.3020803Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:30:56.5731324Z ok (2.392s)
2022-05-18T04:30:56.5731483Z 
2022-05-18T04:30:56.5731958Z ----------------------------------------------------------------------
2022-05-18T04:30:56.5732213Z Ran 1 test in 2.393s
2022-05-18T04:30:56.5732329Z 
2022-05-18T04:30:56.5732391Z OK
2022-05-18T04:30:56.5732485Z 
2022-05-18T04:30:56.5732620Z Generating XML reports...
2022-05-18T04:30:56.5763547Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043054.xml
2022-05-18T04:30:57.5177784Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:30:57.5188114Z 
2022-05-18T04:30:57.5188233Z Running tests...
2022-05-18T04:30:57.5188614Z ----------------------------------------------------------------------
2022-05-18T04:30:57.7993771Z   test_barrier_timeout_global (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26209
2022-05-18T04:30:57.8016060Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26210
2022-05-18T04:30:57.8039044Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26211
2022-05-18T04:30:58.6737960Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:30:58.6838677Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:30:58.6839347Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:30:58.6839971Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:30:58.6840509Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:30:58.6841204Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:30:58.6946386Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:30:58.7852596Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:30:58.7853030Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:30:58.8379563Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:30:58.8380208Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:30:58.8380623Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:30:58.8381483Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:30:58.8382043Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:30:58.8382557Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:31:00.1109006Z ok (2.592s)
2022-05-18T04:31:00.1109278Z 
2022-05-18T04:31:00.1109704Z ----------------------------------------------------------------------
2022-05-18T04:31:00.1110114Z Ran 1 test in 2.592s
2022-05-18T04:31:00.1110295Z 
2022-05-18T04:31:00.1110393Z OK
2022-05-18T04:31:00.1110535Z 
2022-05-18T04:31:00.1110676Z Generating XML reports...
2022-05-18T04:31:00.1141339Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043057.xml
2022-05-18T04:31:01.0574039Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:31:01.0584677Z 
2022-05-18T04:31:01.0585098Z Running tests...
2022-05-18T04:31:01.0585490Z ----------------------------------------------------------------------
2022-05-18T04:31:01.3451520Z   test_barrier_timeout_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26271
2022-05-18T04:31:01.3476950Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26272
2022-05-18T04:31:01.3501545Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26273
2022-05-18T04:31:02.1692802Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:31:02.1693421Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:31:02.1694213Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:31:02.1695337Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:31:02.1714680Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:31:02.1722910Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:31:02.1801614Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:31:02.1802006Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:31:02.1907710Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-05-18T04:31:02.1908400Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:31:02.2706670Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:31:02.2709007Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:31:02.2710012Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:31:02.2717170Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:31:02.2717692Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:31:07.4618799Z ok (6.403s)
2022-05-18T04:31:07.4619016Z 
2022-05-18T04:31:07.4619325Z ----------------------------------------------------------------------
2022-05-18T04:31:07.4619613Z Ran 1 test in 6.403s
2022-05-18T04:31:07.4619758Z 
2022-05-18T04:31:07.4619820Z OK
2022-05-18T04:31:07.4619913Z 
2022-05-18T04:31:07.4619991Z Generating XML reports...
2022-05-18T04:31:07.4651678Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043101.xml
2022-05-18T04:31:08.4111266Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:31:08.4120923Z 
2022-05-18T04:31:08.4121330Z Running tests...
2022-05-18T04:31:08.4121723Z ----------------------------------------------------------------------
2022-05-18T04:31:08.6949559Z   test_batch_isend_irecv_gloo (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26330
2022-05-18T04:31:08.6973864Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26331
2022-05-18T04:31:08.6996146Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26332
2022-05-18T04:31:09.4921235Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:31:09.4972071Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:31:09.4972492Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:31:09.4973101Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:31:09.4973628Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:31:09.5022405Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:31:09.5081733Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:31:09.5082104Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:31:09.6033766Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:31:09.8047583Z ok (1.392s)
2022-05-18T04:31:09.8047817Z 
2022-05-18T04:31:09.8048297Z ----------------------------------------------------------------------
2022-05-18T04:31:09.8048755Z Ran 1 test in 1.393s
2022-05-18T04:31:09.8048914Z 
2022-05-18T04:31:09.8048963Z OK
2022-05-18T04:31:09.8049061Z 
2022-05-18T04:31:09.8049158Z Generating XML reports...
2022-05-18T04:31:09.8079835Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043108.xml
2022-05-18T04:31:10.7306698Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:31:10.7316451Z 
2022-05-18T04:31:10.7316590Z Running tests...
2022-05-18T04:31:10.7317428Z ----------------------------------------------------------------------
2022-05-18T04:31:11.0130582Z   test_batch_isend_irecv_gloo_tags (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26383
2022-05-18T04:31:11.0152802Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26384
2022-05-18T04:31:11.0176051Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26385
2022-05-18T04:31:11.7996273Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:31:11.8096587Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:31:11.8097289Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:31:11.8097963Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:31:11.8098493Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:31:11.8099020Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:31:11.8204487Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:31:11.9110182Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:31:11.9110581Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:31:12.1227544Z ok (1.391s)
2022-05-18T04:31:12.1227804Z 
2022-05-18T04:31:12.1228319Z ----------------------------------------------------------------------
2022-05-18T04:31:12.1228782Z Ran 1 test in 1.391s
2022-05-18T04:31:12.1228999Z 
2022-05-18T04:31:12.1229093Z OK
2022-05-18T04:31:12.1229189Z 
2022-05-18T04:31:12.1229270Z Generating XML reports...
2022-05-18T04:31:12.1260611Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043110.xml
2022-05-18T04:31:13.0587475Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:31:13.0597627Z 
2022-05-18T04:31:13.0597918Z Running tests...
2022-05-18T04:31:13.0598615Z ----------------------------------------------------------------------
2022-05-18T04:31:13.0618557Z   test_batch_isend_irecv_mixed_backend_err (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2022-05-18T04:31:13.0618994Z 
2022-05-18T04:31:13.0619435Z ----------------------------------------------------------------------
2022-05-18T04:31:13.0619753Z Ran 1 test in 0.002s
2022-05-18T04:31:13.0619868Z 
2022-05-18T04:31:13.0619944Z OK (skipped=1)
2022-05-18T04:31:13.0620054Z 
2022-05-18T04:31:13.0620127Z Generating XML reports...
2022-05-18T04:31:13.0650733Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043113.xml
2022-05-18T04:31:13.8954047Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:31:13.8963906Z 
2022-05-18T04:31:13.8964038Z Running tests...
2022-05-18T04:31:13.8964480Z ----------------------------------------------------------------------
2022-05-18T04:31:13.8988117Z   test_batch_isend_irecv_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2022-05-18T04:31:13.8988464Z 
2022-05-18T04:31:13.8988854Z ----------------------------------------------------------------------
2022-05-18T04:31:13.8989109Z Ran 1 test in 0.002s
2022-05-18T04:31:13.8989228Z 
2022-05-18T04:31:13.8989307Z OK (skipped=1)
2022-05-18T04:31:13.8989401Z 
2022-05-18T04:31:13.8989489Z Generating XML reports...
2022-05-18T04:31:13.9027321Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043113.xml
2022-05-18T04:31:14.7329176Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:31:14.7339384Z 
2022-05-18T04:31:14.7339793Z Running tests...
2022-05-18T04:31:14.7340205Z ----------------------------------------------------------------------
2022-05-18T04:31:14.7362710Z   test_batch_isend_irecv_no_rank_zero_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2022-05-18T04:31:14.7363410Z 
2022-05-18T04:31:14.7363950Z ----------------------------------------------------------------------
2022-05-18T04:31:14.7364580Z Ran 1 test in 0.002s
2022-05-18T04:31:14.7364782Z 
2022-05-18T04:31:14.7364916Z OK (skipped=1)
2022-05-18T04:31:14.7365105Z 
2022-05-18T04:31:14.7365248Z Generating XML reports...
2022-05-18T04:31:14.7396132Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043114.xml
2022-05-18T04:31:15.5692849Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:31:15.5703630Z 
2022-05-18T04:31:15.5703877Z Running tests...
2022-05-18T04:31:15.5704484Z ----------------------------------------------------------------------
2022-05-18T04:31:15.5723093Z   test_batch_isend_irecv_op_err (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2022-05-18T04:31:15.5723426Z 
2022-05-18T04:31:15.5723754Z ----------------------------------------------------------------------
2022-05-18T04:31:15.5724011Z Ran 1 test in 0.002s
2022-05-18T04:31:15.5724126Z 
2022-05-18T04:31:15.5724197Z OK (skipped=1)
2022-05-18T04:31:15.5724306Z 
2022-05-18T04:31:15.5724396Z Generating XML reports...
2022-05-18T04:31:15.5755309Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043115.xml
2022-05-18T04:31:16.4043920Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:31:16.4054600Z 
2022-05-18T04:31:16.4054885Z Running tests...
2022-05-18T04:31:16.4055281Z ----------------------------------------------------------------------
2022-05-18T04:31:16.4072293Z   test_batch_isend_irecv_op_list_err (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2022-05-18T04:31:16.4072589Z 
2022-05-18T04:31:16.4072996Z ----------------------------------------------------------------------
2022-05-18T04:31:16.4073467Z Ran 1 test in 0.002s
2022-05-18T04:31:16.4073581Z 
2022-05-18T04:31:16.4073641Z OK (skipped=1)
2022-05-18T04:31:16.4073749Z 
2022-05-18T04:31:16.4073841Z Generating XML reports...
2022-05-18T04:31:16.4105450Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043116.xml
2022-05-18T04:31:17.2385654Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:31:17.2395841Z 
2022-05-18T04:31:17.2395926Z Running tests...
2022-05-18T04:31:17.2396501Z ----------------------------------------------------------------------
2022-05-18T04:31:17.2416982Z   test_batch_isend_irecv_ring_exchange_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2022-05-18T04:31:17.2417487Z 
2022-05-18T04:31:17.2417881Z ----------------------------------------------------------------------
2022-05-18T04:31:17.2418175Z Ran 1 test in 0.002s
2022-05-18T04:31:17.2418303Z 
2022-05-18T04:31:17.2418375Z OK (skipped=1)
2022-05-18T04:31:17.2418487Z 
2022-05-18T04:31:17.2418576Z Generating XML reports...
2022-05-18T04:31:17.2449264Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043117.xml
2022-05-18T04:31:18.0823967Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:31:18.0833890Z 
2022-05-18T04:31:18.0834023Z Running tests...
2022-05-18T04:31:18.0834619Z ----------------------------------------------------------------------
2022-05-18T04:31:18.0855618Z   test_batch_isend_irecv_self_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2022-05-18T04:31:18.0855883Z 
2022-05-18T04:31:18.0856149Z ----------------------------------------------------------------------
2022-05-18T04:31:18.0856457Z Ran 1 test in 0.002s
2022-05-18T04:31:18.0856574Z 
2022-05-18T04:31:18.0856649Z OK (skipped=1)
2022-05-18T04:31:18.0856984Z 
2022-05-18T04:31:18.0857076Z Generating XML reports...
2022-05-18T04:31:18.0887782Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043118.xml
2022-05-18T04:31:18.9226613Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:31:18.9237219Z 
2022-05-18T04:31:18.9237513Z Running tests...
2022-05-18T04:31:18.9237902Z ----------------------------------------------------------------------
2022-05-18T04:31:18.9255233Z   test_batch_isend_irecv_tensor_err (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2022-05-18T04:31:18.9255585Z 
2022-05-18T04:31:18.9255911Z ----------------------------------------------------------------------
2022-05-18T04:31:18.9256190Z Ran 1 test in 0.002s
2022-05-18T04:31:18.9256306Z 
2022-05-18T04:31:18.9256381Z OK (skipped=1)
2022-05-18T04:31:18.9256542Z 
2022-05-18T04:31:18.9256626Z Generating XML reports...
2022-05-18T04:31:18.9292674Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043118.xml
2022-05-18T04:31:19.7656715Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:31:19.7666275Z 
2022-05-18T04:31:19.7666400Z Running tests...
2022-05-18T04:31:19.7667172Z ----------------------------------------------------------------------
2022-05-18T04:31:20.0473321Z   test_broadcast (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26516
2022-05-18T04:31:20.0495320Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26517
2022-05-18T04:31:20.0518248Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26518
2022-05-18T04:31:20.8929660Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:31:20.8982078Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:31:20.8982485Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:31:20.8983247Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:31:20.8983770Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:31:20.9030678Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:31:20.9091131Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:31:20.9091940Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:31:21.0043091Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:31:21.2571777Z ok (1.490s)
2022-05-18T04:31:21.2572038Z 
2022-05-18T04:31:21.2572469Z ----------------------------------------------------------------------
2022-05-18T04:31:21.2572737Z Ran 1 test in 1.490s
2022-05-18T04:31:21.2572854Z 
2022-05-18T04:31:21.2572918Z OK
2022-05-18T04:31:21.2573012Z 
2022-05-18T04:31:21.2573106Z Generating XML reports...
2022-05-18T04:31:21.2603713Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043119.xml
2022-05-18T04:31:22.1778315Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:31:22.1788189Z 
2022-05-18T04:31:22.1788391Z Running tests...
2022-05-18T04:31:22.1789025Z ----------------------------------------------------------------------
2022-05-18T04:31:22.4662577Z   test_broadcast_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26572
2022-05-18T04:31:22.4684094Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26573
2022-05-18T04:31:22.4706499Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26574
2022-05-18T04:31:23.2779069Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:31:23.2853217Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:31:23.2853633Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:31:23.2854244Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:31:23.2854773Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:31:23.2879842Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:31:23.2962536Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:31:23.2963087Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:31:23.3892756Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:31:23.5759446Z skip: CUDA is not available. (1.397s)
2022-05-18T04:31:23.5759714Z 
2022-05-18T04:31:23.5760154Z ----------------------------------------------------------------------
2022-05-18T04:31:23.5760559Z Ran 1 test in 1.397s
2022-05-18T04:31:23.5760724Z 
2022-05-18T04:31:23.5760836Z OK (skipped=1)
2022-05-18T04:31:23.5760999Z 
2022-05-18T04:31:23.5761142Z Generating XML reports...
2022-05-18T04:31:23.5792552Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043122.xml
2022-05-18T04:31:24.4912752Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:31:24.4921806Z 
2022-05-18T04:31:24.4922069Z Running tests...
2022-05-18T04:31:24.4922674Z ----------------------------------------------------------------------
2022-05-18T04:31:24.7723580Z   test_broadcast_full_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26625
2022-05-18T04:31:24.7745651Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26626
2022-05-18T04:31:24.7768543Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26627
2022-05-18T04:31:25.5800825Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:31:25.5902321Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:31:25.5903166Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:31:25.5903803Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:31:25.5904348Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:31:25.5904873Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:31:25.5912246Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:31:25.5913605Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:31:25.5914163Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:31:25.6220158Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:31:25.6220794Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:31:25.6221618Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-05-18T04:31:25.6222592Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:31:25.6223384Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:31:25.6223910Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:31:25.8819477Z ok (1.389s)
2022-05-18T04:31:25.8819643Z 
2022-05-18T04:31:25.8820036Z ----------------------------------------------------------------------
2022-05-18T04:31:25.8820318Z Ran 1 test in 1.390s
2022-05-18T04:31:25.8820435Z 
2022-05-18T04:31:25.8820506Z OK
2022-05-18T04:31:25.8820599Z 
2022-05-18T04:31:25.8820692Z Generating XML reports...
2022-05-18T04:31:25.8851758Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043124.xml
2022-05-18T04:31:26.8063455Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:31:26.8073391Z 
2022-05-18T04:31:26.8073547Z Running tests...
2022-05-18T04:31:26.8074152Z ----------------------------------------------------------------------
2022-05-18T04:31:27.0887144Z   test_broadcast_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26690
2022-05-18T04:31:27.0908614Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26691
2022-05-18T04:31:27.0931174Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26692
2022-05-18T04:31:27.9309344Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:31:27.9310336Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:31:27.9310987Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:31:27.9311614Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:31:27.9312460Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:31:27.9313069Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:31:27.9415997Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:31:28.0323124Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:31:28.0324077Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:31:28.0324865Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:31:28.0429798Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-05-18T04:31:28.0430394Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:31:28.0431290Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:31:28.0432165Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:31:28.0528680Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:31:28.3983764Z ok (1.591s)
2022-05-18T04:31:28.3983963Z 
2022-05-18T04:31:28.3984337Z ----------------------------------------------------------------------
2022-05-18T04:31:28.3984694Z Ran 1 test in 1.591s
2022-05-18T04:31:28.3985023Z 
2022-05-18T04:31:28.3985087Z OK
2022-05-18T04:31:28.3985180Z 
2022-05-18T04:31:28.3985278Z Generating XML reports...
2022-05-18T04:31:28.4015597Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043126.xml
2022-05-18T04:31:29.3285295Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:31:29.3295315Z 
2022-05-18T04:31:29.3295453Z Running tests...
2022-05-18T04:31:29.3296052Z ----------------------------------------------------------------------
2022-05-18T04:31:29.6118381Z   test_broadcast_multigpu (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26751
2022-05-18T04:31:29.6140855Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26752
2022-05-18T04:31:29.6163157Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26753
2022-05-18T04:31:30.4532723Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:31:30.4533489Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:31:30.4533965Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:31:30.4534575Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:31:30.4535103Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:31:30.4535622Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:31:30.4639254Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:31:30.5546925Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:31:30.5547466Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:31:30.7213619Z skip: CUDA is not available. (1.392s)
2022-05-18T04:31:30.7213899Z 
2022-05-18T04:31:30.7214417Z ----------------------------------------------------------------------
2022-05-18T04:31:30.7214840Z Ran 1 test in 1.392s
2022-05-18T04:31:30.7214958Z 
2022-05-18T04:31:30.7215037Z OK (skipped=1)
2022-05-18T04:31:30.7215145Z 
2022-05-18T04:31:30.7215220Z Generating XML reports...
2022-05-18T04:31:30.7246033Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043129.xml
2022-05-18T04:31:31.6474721Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:31:31.6484597Z 
2022-05-18T04:31:31.6484729Z Running tests...
2022-05-18T04:31:31.6485332Z ----------------------------------------------------------------------
2022-05-18T04:31:31.9278473Z   test_broadcast_object_list (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26804
2022-05-18T04:31:31.9300740Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26805
2022-05-18T04:31:31.9323603Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26806
2022-05-18T04:31:32.7580252Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:31:32.7681892Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:31:32.7682302Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:31:32.7682926Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:31:32.7683448Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:31:32.7782722Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:31:32.7793281Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:31:32.7793857Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:31:32.7794410Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:31:32.9373173Z ok (1.289s)
2022-05-18T04:31:32.9373403Z 
2022-05-18T04:31:32.9373840Z ----------------------------------------------------------------------
2022-05-18T04:31:32.9374169Z Ran 1 test in 1.289s
2022-05-18T04:31:32.9374288Z 
2022-05-18T04:31:32.9374336Z OK
2022-05-18T04:31:32.9374432Z 
2022-05-18T04:31:32.9374527Z Generating XML reports...
2022-05-18T04:31:32.9405409Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043131.xml
2022-05-18T04:31:33.8590572Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:31:33.8600674Z 
2022-05-18T04:31:33.8600799Z Running tests...
2022-05-18T04:31:33.8601338Z ----------------------------------------------------------------------
2022-05-18T04:31:33.8616128Z   test_compute_bucket_assignment_by_size_sparse_error_with_logger (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'nccl'} (0.001s)
2022-05-18T04:31:33.8616641Z 
2022-05-18T04:31:33.8617025Z ----------------------------------------------------------------------
2022-05-18T04:31:33.8617317Z Ran 1 test in 0.002s
2022-05-18T04:31:33.8617418Z 
2022-05-18T04:31:33.8617490Z OK (skipped=1)
2022-05-18T04:31:33.8617598Z 
2022-05-18T04:31:33.8617682Z Generating XML reports...
2022-05-18T04:31:33.8648263Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043133.xml
2022-05-18T04:31:34.6940321Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:31:34.6950765Z 
2022-05-18T04:31:34.6950892Z Running tests...
2022-05-18T04:31:34.6951486Z ----------------------------------------------------------------------
2022-05-18T04:31:34.6966956Z   test_compute_bucket_assignment_by_size_sparse_error_without_logger (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'nccl', 'gloo'} (0.001s)
2022-05-18T04:31:34.6967249Z 
2022-05-18T04:31:34.6967454Z ----------------------------------------------------------------------
2022-05-18T04:31:34.6967700Z Ran 1 test in 0.002s
2022-05-18T04:31:34.6967800Z 
2022-05-18T04:31:34.6967878Z OK (skipped=1)
2022-05-18T04:31:34.6967985Z 
2022-05-18T04:31:34.6968072Z Generating XML reports...
2022-05-18T04:31:34.6998824Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043134.xml
2022-05-18T04:31:35.5279788Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:31:35.5289527Z 
2022-05-18T04:31:35.5289624Z Running tests...
2022-05-18T04:31:35.5290930Z ----------------------------------------------------------------------
2022-05-18T04:31:35.8124332Z   test_ddp_broadcast_buffer (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26877
2022-05-18T04:31:35.8146248Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26878
2022-05-18T04:31:35.8169189Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26879
2022-05-18T04:31:36.6332703Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:31:36.6333430Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:31:36.6333970Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:31:36.6334766Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:31:36.6335350Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:31:36.6335873Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:31:36.6441117Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:31:36.6441711Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:31:36.7347110Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:31:36.9220543Z skip: Need at least 2 CUDA devices (1.393s)
2022-05-18T04:31:36.9220918Z 
2022-05-18T04:31:36.9221385Z ----------------------------------------------------------------------
2022-05-18T04:31:36.9221653Z Ran 1 test in 1.393s
2022-05-18T04:31:36.9221768Z 
2022-05-18T04:31:36.9221842Z OK (skipped=1)
2022-05-18T04:31:36.9221948Z 
2022-05-18T04:31:36.9222027Z Generating XML reports...
2022-05-18T04:31:36.9252511Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043135.xml
2022-05-18T04:31:37.8471695Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:31:37.8482534Z 
2022-05-18T04:31:37.8483018Z Running tests...
2022-05-18T04:31:37.8483401Z ----------------------------------------------------------------------
2022-05-18T04:31:38.1287632Z   test_ddp_broadcast_buffer_via_hook (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26930
2022-05-18T04:31:38.1309533Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26931
2022-05-18T04:31:38.1332888Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26932
2022-05-18T04:31:38.9515951Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:31:38.9616255Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:31:38.9616664Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:31:38.9617266Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:31:38.9619229Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:31:38.9619757Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:31:38.9725449Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:31:38.9725964Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:31:39.0631008Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:31:39.2383691Z skip: Need at least 2 CUDA devices (1.390s)
2022-05-18T04:31:39.2383916Z 
2022-05-18T04:31:39.2384258Z ----------------------------------------------------------------------
2022-05-18T04:31:39.2384527Z Ran 1 test in 1.390s
2022-05-18T04:31:39.2384686Z 
2022-05-18T04:31:39.2384763Z OK (skipped=1)
2022-05-18T04:31:39.2384873Z 
2022-05-18T04:31:39.2384958Z Generating XML reports...
2022-05-18T04:31:39.2416918Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043137.xml
2022-05-18T04:31:40.1615829Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:31:40.1625590Z 
2022-05-18T04:31:40.1625731Z Running tests...
2022-05-18T04:31:40.1626458Z ----------------------------------------------------------------------
2022-05-18T04:31:40.4455021Z   test_ddp_buffer_hook_allreduce (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26983
2022-05-18T04:31:40.4477155Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26984
2022-05-18T04:31:40.4499705Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26985
2022-05-18T04:31:41.2384645Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:31:41.2485665Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:31:41.2486625Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:31:41.2487284Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:31:41.2487893Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:31:41.2488446Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:31:41.2495367Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:31:41.2496049Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:31:41.2497138Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:31:41.4549786Z skip: Need at least 2 CUDA devices (1.292s)
2022-05-18T04:31:41.4549968Z 
2022-05-18T04:31:41.4550393Z ----------------------------------------------------------------------
2022-05-18T04:31:41.4550701Z Ran 1 test in 1.292s
2022-05-18T04:31:41.4550806Z 
2022-05-18T04:31:41.4550880Z OK (skipped=1)
2022-05-18T04:31:41.4551002Z 
2022-05-18T04:31:41.4551089Z Generating XML reports...
2022-05-18T04:31:41.4582062Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043140.xml
2022-05-18T04:31:42.3774403Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:31:42.3784282Z 
2022-05-18T04:31:42.3784769Z Running tests...
2022-05-18T04:31:42.3785349Z ----------------------------------------------------------------------
2022-05-18T04:31:42.6514363Z   test_ddp_buffer_hook_allreduce_return_future (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77261 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure IN_CI is not set and you are not using the flag --import-disabled-tests. (0.273s)
2022-05-18T04:31:42.6515429Z 
2022-05-18T04:31:42.6515785Z ----------------------------------------------------------------------
2022-05-18T04:31:42.6516099Z Ran 1 test in 0.273s
2022-05-18T04:31:42.6516225Z 
2022-05-18T04:31:42.6516300Z OK (skipped=1)
2022-05-18T04:31:42.6516409Z 
2022-05-18T04:31:42.6516499Z Generating XML reports...
2022-05-18T04:31:42.6542485Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043142.xml
2022-05-18T04:31:43.5530261Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:31:43.5540244Z 
2022-05-18T04:31:43.5540691Z Running tests...
2022-05-18T04:31:43.5541110Z ----------------------------------------------------------------------
2022-05-18T04:31:43.5570999Z   test_ddp_build_debug_param_to_name_mapping (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'nccl'} (0.003s)
2022-05-18T04:31:43.5571379Z 
2022-05-18T04:31:43.5571635Z ----------------------------------------------------------------------
2022-05-18T04:31:43.5572115Z Ran 1 test in 0.003s
2022-05-18T04:31:43.5572231Z 
2022-05-18T04:31:43.5572306Z OK (skipped=1)
2022-05-18T04:31:43.5572414Z 
2022-05-18T04:31:43.5572499Z Generating XML reports...
2022-05-18T04:31:43.5603732Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043143.xml
2022-05-18T04:31:44.4047087Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:31:44.4057517Z 
2022-05-18T04:31:44.4057904Z Running tests...
2022-05-18T04:31:44.4058309Z ----------------------------------------------------------------------
2022-05-18T04:31:44.6969084Z   test_ddp_build_debug_param_to_name_mapping_requires_grad (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27056
2022-05-18T04:31:44.6991319Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27057
2022-05-18T04:31:44.7014862Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27058
2022-05-18T04:31:45.5280256Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:31:45.5380873Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:31:45.5381333Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:31:45.5382363Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:31:45.5383108Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:31:45.5383652Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:31:45.5489252Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:31:45.6397427Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:31:45.6398067Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:31:45.8066786Z skip: Need at least 2 CUDA devices (1.400s)
2022-05-18T04:31:45.8067203Z 
2022-05-18T04:31:45.8067504Z ----------------------------------------------------------------------
2022-05-18T04:31:45.8067858Z Ran 1 test in 1.401s
2022-05-18T04:31:45.8068247Z 
2022-05-18T04:31:45.8068322Z OK (skipped=1)
2022-05-18T04:31:45.8068461Z 
2022-05-18T04:31:45.8068591Z Generating XML reports...
2022-05-18T04:31:45.8099382Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043144.xml
2022-05-18T04:31:46.7494201Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:31:46.7504427Z 
2022-05-18T04:31:46.7504537Z Running tests...
2022-05-18T04:31:46.7504972Z ----------------------------------------------------------------------
2022-05-18T04:31:47.0370517Z   test_ddp_comm_hook_logging (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27109
2022-05-18T04:31:47.0392539Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27110
2022-05-18T04:31:47.0415183Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27111
2022-05-18T04:31:47.8616660Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:31:47.8717556Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:31:47.8718243Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:31:47.8719021Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:31:47.8719746Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:31:47.8720329Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:31:47.8728128Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:31:47.8728774Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:31:47.8729987Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:31:48.0463960Z skip: Need at least 3 CUDA devices (1.296s)
2022-05-18T04:31:48.0464220Z 
2022-05-18T04:31:48.0464549Z ----------------------------------------------------------------------
2022-05-18T04:31:48.0464788Z Ran 1 test in 1.296s
2022-05-18T04:31:48.0464903Z 
2022-05-18T04:31:48.0464978Z OK (skipped=1)
2022-05-18T04:31:48.0465103Z 
2022-05-18T04:31:48.0465191Z Generating XML reports...
2022-05-18T04:31:48.0496484Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043146.xml
2022-05-18T04:31:48.9712135Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:31:48.9721841Z 
2022-05-18T04:31:48.9721922Z Running tests...
2022-05-18T04:31:48.9722408Z ----------------------------------------------------------------------
2022-05-18T04:31:48.9761341Z   test_ddp_control_flow_different_across_ranks (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'nccl'} (0.004s)
2022-05-18T04:31:48.9761800Z 
2022-05-18T04:31:48.9762189Z ----------------------------------------------------------------------
2022-05-18T04:31:48.9762517Z Ran 1 test in 0.004s
2022-05-18T04:31:48.9762620Z 
2022-05-18T04:31:48.9762694Z OK (skipped=1)
2022-05-18T04:31:48.9762802Z 
2022-05-18T04:31:48.9762889Z Generating XML reports...
2022-05-18T04:31:48.9793273Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043148.xml
2022-05-18T04:31:49.8159253Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:31:49.8169579Z 
2022-05-18T04:31:49.8169700Z Running tests...
2022-05-18T04:31:49.8170252Z ----------------------------------------------------------------------
2022-05-18T04:31:49.8203155Z   test_ddp_control_flow_same_across_ranks (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'nccl', 'gloo'} (0.003s)
2022-05-18T04:31:49.8203574Z 
2022-05-18T04:31:49.8203823Z ----------------------------------------------------------------------
2022-05-18T04:31:49.8204055Z Ran 1 test in 0.003s
2022-05-18T04:31:49.8204170Z 
2022-05-18T04:31:49.8204245Z OK (skipped=1)
2022-05-18T04:31:49.8204353Z 
2022-05-18T04:31:49.8204440Z Generating XML reports...
2022-05-18T04:31:49.8235994Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043149.xml
2022-05-18T04:31:50.6532195Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:31:50.6541925Z 
2022-05-18T04:31:50.6542045Z Running tests...
2022-05-18T04:31:50.6542652Z ----------------------------------------------------------------------
2022-05-18T04:31:50.9340786Z   test_ddp_create_graph (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27182
2022-05-18T04:31:50.9362730Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27183
2022-05-18T04:31:50.9385824Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27184
2022-05-18T04:31:51.7797904Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:31:51.7899369Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:31:51.7900508Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:31:51.7901011Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:31:51.7901495Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:31:51.7902016Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:31:51.7910453Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:31:51.7911781Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:31:51.7912736Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:31:51.7974674Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2_u6pid5
2022-05-18T04:31:51.7976014Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2_u6pid5/_remote_module_non_scriptable.py
2022-05-18T04:31:51.7976698Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa64i9hpg
2022-05-18T04:31:51.7977130Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppvuek6jy
2022-05-18T04:31:51.7978948Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa64i9hpg/_remote_module_non_scriptable.py
2022-05-18T04:31:51.7979634Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppvuek6jy/_remote_module_non_scriptable.py
2022-05-18T04:31:51.8086551Z [W reducer.cpp:367] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-05-18T04:31:51.8088071Z [W reducer.cpp:367] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-05-18T04:31:51.8089908Z [W reducer.cpp:367] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-05-18T04:31:51.8092443Z /opt/conda/lib/python3.7/site-packages/torch/autograd/__init__.py:175: UserWarning: Using backward() with create_graph=True will create a reference cycle between the parameter and its gradient which can cause a memory leak. We recommend using autograd.grad when creating the graph to avoid this. If you have to use this function, make sure to reset the .grad fields of your parameters to None after use to break the cycle and avoid the leak. (Triggered internally at  /var/lib/jenkins/workspace/torch/csrc/autograd/engine.cpp:995.)
2022-05-18T04:31:51.8093499Z   allow_unreachable=True, accumulate_grad=True)  # Calls into the C++ engine to run the backward pass
2022-05-18T04:31:51.8094763Z /opt/conda/lib/python3.7/site-packages/torch/autograd/__init__.py:175: UserWarning: Using backward() with create_graph=True will create a reference cycle between the parameter and its gradient which can cause a memory leak. We recommend using autograd.grad when creating the graph to avoid this. If you have to use this function, make sure to reset the .grad fields of your parameters to None after use to break the cycle and avoid the leak. (Triggered internally at  /var/lib/jenkins/workspace/torch/csrc/autograd/engine.cpp:995.)
2022-05-18T04:31:51.8096167Z   allow_unreachable=True, accumulate_grad=True)  # Calls into the C++ engine to run the backward pass
2022-05-18T04:31:51.8098108Z /opt/conda/lib/python3.7/site-packages/torch/autograd/__init__.py:175: UserWarning: Using backward() with create_graph=True will create a reference cycle between the parameter and its gradient which can cause a memory leak. We recommend using autograd.grad when creating the graph to avoid this. If you have to use this function, make sure to reset the .grad fields of your parameters to None after use to break the cycle and avoid the leak. (Triggered internally at  /var/lib/jenkins/workspace/torch/csrc/autograd/engine.cpp:995.)
2022-05-18T04:31:51.8099223Z   allow_unreachable=True, accumulate_grad=True)  # Calls into the C++ engine to run the backward pass
2022-05-18T04:31:51.8099809Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:31:51.8100384Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:31:51.8100929Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:31:51.8102391Z [W reducer.cpp:367] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-05-18T04:31:51.8104244Z [W reducer.cpp:367] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-05-18T04:31:51.8105705Z [W reducer.cpp:367] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-05-18T04:31:51.8107469Z [W reducer.cpp:367] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-05-18T04:31:51.8109338Z [W reducer.cpp:367] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-05-18T04:31:51.8110857Z [W reducer.cpp:367] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-05-18T04:31:51.8112429Z [W reducer.cpp:367] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-05-18T04:31:51.8114532Z [W reducer.cpp:367] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-05-18T04:31:51.8116649Z [W reducer.cpp:367] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-05-18T04:31:51.8117947Z [W reducer.cpp:367] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-05-18T04:31:51.8119079Z [W reducer.cpp:367] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-05-18T04:31:51.8120330Z [W reducer.cpp:367] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-05-18T04:31:51.8121909Z [W reducer.cpp:367] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-05-18T04:31:51.8123116Z [W reducer.cpp:367] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-05-18T04:31:51.8124483Z [W reducer.cpp:367] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-05-18T04:31:51.9433888Z ok (1.289s)
2022-05-18T04:31:51.9434166Z 
2022-05-18T04:31:51.9434711Z ----------------------------------------------------------------------
2022-05-18T04:31:51.9435036Z Ran 1 test in 1.289s
2022-05-18T04:31:51.9435159Z 
2022-05-18T04:31:51.9435225Z OK
2022-05-18T04:31:51.9435304Z 
2022-05-18T04:31:51.9435398Z Generating XML reports...
2022-05-18T04:31:51.9465897Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043150.xml
2022-05-18T04:31:52.8698250Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:31:52.8708516Z 
2022-05-18T04:31:52.8708738Z Running tests...
2022-05-18T04:31:52.8709068Z ----------------------------------------------------------------------
2022-05-18T04:31:52.8753354Z   test_ddp_device (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'nccl', 'gloo'} (0.004s)
2022-05-18T04:31:52.8754063Z 
2022-05-18T04:31:52.8754288Z ----------------------------------------------------------------------
2022-05-18T04:31:52.8754537Z Ran 1 test in 0.004s
2022-05-18T04:31:52.8754652Z 
2022-05-18T04:31:52.8754780Z OK (skipped=1)
2022-05-18T04:31:52.8754889Z 
2022-05-18T04:31:52.8754975Z Generating XML reports...
2022-05-18T04:31:52.8784920Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043152.xml
2022-05-18T04:31:53.7160680Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:31:53.7171039Z 
2022-05-18T04:31:53.7171172Z Running tests...
2022-05-18T04:31:53.7171749Z ----------------------------------------------------------------------
2022-05-18T04:31:53.7199646Z   test_ddp_forward_backward_hook (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'nccl', 'gloo'} (0.003s)
2022-05-18T04:31:53.7199948Z 
2022-05-18T04:31:53.7200167Z ----------------------------------------------------------------------
2022-05-18T04:31:53.7200425Z Ran 1 test in 0.003s
2022-05-18T04:31:53.7200539Z 
2022-05-18T04:31:53.7200614Z OK (skipped=1)
2022-05-18T04:31:53.7200708Z 
2022-05-18T04:31:53.7200801Z Generating XML reports...
2022-05-18T04:31:53.7233077Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043153.xml
2022-05-18T04:31:54.5547343Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:31:54.5558203Z 
2022-05-18T04:31:54.5558608Z Running tests...
2022-05-18T04:31:54.5559061Z ----------------------------------------------------------------------
2022-05-18T04:31:54.8392846Z   test_ddp_grad_div_uneven_inputs (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27255
2022-05-18T04:31:54.8415084Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27256
2022-05-18T04:31:54.8437928Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27257
2022-05-18T04:31:55.6523232Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:31:55.6523745Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:31:55.6524100Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:31:55.6524722Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:31:55.6525248Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:31:55.6525769Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:31:55.6632321Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:31:55.6633066Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:31:55.7536090Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:31:55.9489905Z skip: Need at least 2 CUDA devices (1.393s)
2022-05-18T04:31:55.9490116Z 
2022-05-18T04:31:55.9490485Z ----------------------------------------------------------------------
2022-05-18T04:31:55.9490755Z Ran 1 test in 1.393s
2022-05-18T04:31:55.9490868Z 
2022-05-18T04:31:55.9490943Z OK (skipped=1)
2022-05-18T04:31:55.9491067Z 
2022-05-18T04:31:55.9491187Z Generating XML reports...
2022-05-18T04:31:55.9522069Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043154.xml
2022-05-18T04:31:56.8801643Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:31:56.8811433Z 
2022-05-18T04:31:56.8811790Z Running tests...
2022-05-18T04:31:56.8812398Z ----------------------------------------------------------------------
2022-05-18T04:31:57.1558334Z   test_ddp_hook_parity_allreduce (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77293 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure IN_CI is not set and you are not using the flag --import-disabled-tests. (0.274s)
2022-05-18T04:31:57.1558845Z 
2022-05-18T04:31:57.1559056Z ----------------------------------------------------------------------
2022-05-18T04:31:57.1559315Z Ran 1 test in 0.274s
2022-05-18T04:31:57.1559429Z 
2022-05-18T04:31:57.1559502Z OK (skipped=1)
2022-05-18T04:31:57.1559596Z 
2022-05-18T04:31:57.1559683Z Generating XML reports...
2022-05-18T04:31:57.1586575Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043156.xml
2022-05-18T04:31:58.0678780Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:31:58.0688718Z 
2022-05-18T04:31:58.0689203Z Running tests...
2022-05-18T04:31:58.0689878Z ----------------------------------------------------------------------
2022-05-18T04:31:58.3496264Z   test_ddp_hook_parity_allreduce_process_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27318
2022-05-18T04:31:58.3518690Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27319
2022-05-18T04:31:58.3540780Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27320
2022-05-18T04:31:59.1426480Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:31:59.1525252Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:31:59.1525757Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:31:59.1526465Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:31:59.1527122Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:31:59.1527656Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:31:59.1635459Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:31:59.1636053Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:31:59.2540884Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:31:59.4592046Z skip: Need at least 3 CUDA devices (1.390s)
2022-05-18T04:31:59.4592318Z 
2022-05-18T04:31:59.4592849Z ----------------------------------------------------------------------
2022-05-18T04:31:59.4593318Z Ran 1 test in 1.390s
2022-05-18T04:31:59.4593532Z 
2022-05-18T04:31:59.4593653Z OK (skipped=1)
2022-05-18T04:31:59.4593776Z 
2022-05-18T04:31:59.4593869Z Generating XML reports...
2022-05-18T04:31:59.4623948Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043158.xml
2022-05-18T04:32:00.4023872Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:32:00.4034710Z 
2022-05-18T04:32:00.4035151Z Running tests...
2022-05-18T04:32:00.4035776Z ----------------------------------------------------------------------
2022-05-18T04:32:00.6905194Z   test_ddp_hook_parity_post_localSGD (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27371
2022-05-18T04:32:00.6928394Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27372
2022-05-18T04:32:00.6951457Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27373
2022-05-18T04:32:01.5484610Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:32:01.5586176Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:32:01.5586797Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:32:01.5587728Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:32:01.5588601Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:32:01.5589368Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:32:01.5597834Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:32:01.5598815Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:32:01.5599380Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:32:01.8001670Z skip: Need at least 3 CUDA devices (1.396s)
2022-05-18T04:32:01.8001930Z 
2022-05-18T04:32:01.8002333Z ----------------------------------------------------------------------
2022-05-18T04:32:01.8002605Z Ran 1 test in 1.397s
2022-05-18T04:32:01.8002720Z 
2022-05-18T04:32:01.8002794Z OK (skipped=1)
2022-05-18T04:32:01.8002903Z 
2022-05-18T04:32:01.8002975Z Generating XML reports...
2022-05-18T04:32:01.8034098Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043200.xml
2022-05-18T04:32:02.7520488Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:32:02.7529577Z 
2022-05-18T04:32:02.7529893Z Running tests...
2022-05-18T04:32:02.7530268Z ----------------------------------------------------------------------
2022-05-18T04:32:03.0393894Z   test_ddp_hook_parity_powerSGD (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27424
2022-05-18T04:32:03.0415381Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27425
2022-05-18T04:32:03.0438490Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27426
2022-05-18T04:32:03.8626231Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:32:03.8660348Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:32:03.8660805Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:32:03.8661449Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:32:03.8661998Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:32:03.8728143Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:32:03.8770343Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:32:03.8770946Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:32:03.9741300Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:32:04.1490130Z skip: Need at least 3 CUDA devices (1.396s)
2022-05-18T04:32:04.1490465Z 
2022-05-18T04:32:04.1490923Z ----------------------------------------------------------------------
2022-05-18T04:32:04.1491308Z Ran 1 test in 1.396s
2022-05-18T04:32:04.1491479Z 
2022-05-18T04:32:04.1491595Z OK (skipped=1)
2022-05-18T04:32:04.1492016Z 
2022-05-18T04:32:04.1492136Z Generating XML reports...
2022-05-18T04:32:04.1523979Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043202.xml
2022-05-18T04:32:05.1210774Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:32:05.1220630Z 
2022-05-18T04:32:05.1220776Z Running tests...
2022-05-18T04:32:05.1221336Z ----------------------------------------------------------------------
2022-05-18T04:32:05.4133401Z   test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27477
2022-05-18T04:32:05.4156659Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27478
2022-05-18T04:32:05.4180506Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27479
2022-05-18T04:32:06.2185956Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:32:06.2186735Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:32:06.2187419Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:32:06.2188196Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:32:06.2188725Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:32:06.2189251Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:32:06.2292522Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:32:06.3199851Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:32:06.3200526Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:32:06.5233098Z skip: Need at least 2 CUDA devices (1.401s)
2022-05-18T04:32:06.5233417Z 
2022-05-18T04:32:06.5233922Z ----------------------------------------------------------------------
2022-05-18T04:32:06.5234255Z Ran 1 test in 1.401s
2022-05-18T04:32:06.5234373Z 
2022-05-18T04:32:06.5234448Z OK (skipped=1)
2022-05-18T04:32:06.5234557Z 
2022-05-18T04:32:06.5234644Z Generating XML reports...
2022-05-18T04:32:06.5267093Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043205.xml
2022-05-18T04:32:07.4690555Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:32:07.4700404Z 
2022-05-18T04:32:07.4700555Z Running tests...
2022-05-18T04:32:07.4701148Z ----------------------------------------------------------------------
2022-05-18T04:32:07.7556219Z   test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27530
2022-05-18T04:32:07.7578582Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27531
2022-05-18T04:32:07.7601792Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27532
2022-05-18T04:32:08.5756541Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:32:08.5757147Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:32:08.5757535Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:32:08.5758143Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:32:08.5758678Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:32:08.5759444Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:32:08.5863964Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:32:08.6772067Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:32:08.6772631Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:32:08.8653446Z skip: Need at least 2 CUDA devices (1.395s)
2022-05-18T04:32:08.8653621Z 
2022-05-18T04:32:08.8653953Z ----------------------------------------------------------------------
2022-05-18T04:32:08.8654209Z Ran 1 test in 1.395s
2022-05-18T04:32:08.8654327Z 
2022-05-18T04:32:08.8654401Z OK (skipped=1)
2022-05-18T04:32:08.8654508Z 
2022-05-18T04:32:08.8654596Z Generating XML reports...
2022-05-18T04:32:08.8686027Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043207.xml
2022-05-18T04:32:09.8155800Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:32:09.8165864Z 
2022-05-18T04:32:09.8165974Z Running tests...
2022-05-18T04:32:09.8166979Z ----------------------------------------------------------------------
2022-05-18T04:32:10.1022854Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27583
2022-05-18T04:32:10.1043959Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27584
2022-05-18T04:32:10.1066765Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27585
2022-05-18T04:32:10.9254327Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:32:10.9254886Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:32:10.9255275Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:32:10.9255896Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:32:10.9256433Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:32:10.9256949Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:32:10.9265906Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:32:11.0267590Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:32:11.0268180Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:32:11.2119859Z skip: Need at least 2 CUDA devices (1.395s)
2022-05-18T04:32:11.2120215Z 
2022-05-18T04:32:11.2120751Z ----------------------------------------------------------------------
2022-05-18T04:32:11.2121049Z Ran 1 test in 1.395s
2022-05-18T04:32:11.2121164Z 
2022-05-18T04:32:11.2121237Z OK (skipped=1)
2022-05-18T04:32:11.2121332Z 
2022-05-18T04:32:11.2121416Z Generating XML reports...
2022-05-18T04:32:11.2153886Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043209.xml
2022-05-18T04:32:12.1484923Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:32:12.1495074Z 
2022-05-18T04:32:12.1495560Z Running tests...
2022-05-18T04:32:12.1495976Z ----------------------------------------------------------------------
2022-05-18T04:32:12.4382213Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27636
2022-05-18T04:32:12.4404907Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27637
2022-05-18T04:32:12.4427167Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27638
2022-05-18T04:32:13.2730700Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:32:13.2832121Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:32:13.2833257Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:32:13.2833807Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:32:13.2834290Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:32:13.2834832Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:32:13.2842032Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:32:13.2843162Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:32:13.2844369Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:32:13.4479162Z skip: Need at least 2 CUDA devices (1.298s)
2022-05-18T04:32:13.4479398Z 
2022-05-18T04:32:13.4479702Z ----------------------------------------------------------------------
2022-05-18T04:32:13.4479958Z Ran 1 test in 1.298s
2022-05-18T04:32:13.4480111Z 
2022-05-18T04:32:13.4480186Z OK (skipped=1)
2022-05-18T04:32:13.4480295Z 
2022-05-18T04:32:13.4480383Z Generating XML reports...
2022-05-18T04:32:13.4510931Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043212.xml
2022-05-18T04:32:14.3711565Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:32:14.3721664Z 
2022-05-18T04:32:14.3722190Z Running tests...
2022-05-18T04:32:14.3722730Z ----------------------------------------------------------------------
2022-05-18T04:32:14.6538826Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27689
2022-05-18T04:32:14.6560317Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27690
2022-05-18T04:32:14.6583114Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27691
2022-05-18T04:32:15.4556638Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:32:15.4657047Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:32:15.4657730Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:32:15.4658587Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:32:15.4659108Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:32:15.4659632Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:32:15.4767041Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:32:15.4767971Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:32:15.5671856Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:32:15.7636208Z skip: Need at least 2 CUDA devices (1.391s)
2022-05-18T04:32:15.7636571Z 
2022-05-18T04:32:15.7637256Z ----------------------------------------------------------------------
2022-05-18T04:32:15.7637644Z Ran 1 test in 1.391s
2022-05-18T04:32:15.7637790Z 
2022-05-18T04:32:15.7637884Z OK (skipped=1)
2022-05-18T04:32:15.7638031Z 
2022-05-18T04:32:15.7638150Z Generating XML reports...
2022-05-18T04:32:15.7669254Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043214.xml
2022-05-18T04:32:16.6869245Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:32:16.6879002Z 
2022-05-18T04:32:16.6879137Z Running tests...
2022-05-18T04:32:16.6879580Z ----------------------------------------------------------------------
2022-05-18T04:32:16.9700842Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27742
2022-05-18T04:32:16.9722116Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27743
2022-05-18T04:32:16.9745487Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27744
2022-05-18T04:32:17.7842237Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:32:17.7943339Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:32:17.7944035Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:32:17.7944665Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:32:17.7945189Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:32:17.7945721Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:32:17.7954419Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:32:17.7955253Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:32:17.7956406Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:32:17.9794424Z skip: Need at least 2 CUDA devices (1.291s)
2022-05-18T04:32:17.9794704Z 
2022-05-18T04:32:17.9795025Z ----------------------------------------------------------------------
2022-05-18T04:32:17.9795261Z Ran 1 test in 1.291s
2022-05-18T04:32:17.9795373Z 
2022-05-18T04:32:17.9795471Z OK (skipped=1)
2022-05-18T04:32:17.9795612Z 
2022-05-18T04:32:17.9795698Z Generating XML reports...
2022-05-18T04:32:17.9826465Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043216.xml
2022-05-18T04:32:18.9009615Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:32:18.9018999Z 
2022-05-18T04:32:18.9019141Z Running tests...
2022-05-18T04:32:18.9019740Z ----------------------------------------------------------------------
2022-05-18T04:32:19.1837706Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27795
2022-05-18T04:32:19.1859884Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27796
2022-05-18T04:32:19.1883298Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27797
2022-05-18T04:32:19.9779397Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:32:19.9780145Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:32:19.9780736Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:32:19.9781413Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:32:19.9781947Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:32:19.9782469Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:32:19.9789687Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:32:19.9790671Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:32:19.9791100Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:32:20.1933805Z skip: Need at least 2 CUDA devices (1.291s)
2022-05-18T04:32:20.1934034Z 
2022-05-18T04:32:20.1934343Z ----------------------------------------------------------------------
2022-05-18T04:32:20.1934613Z Ran 1 test in 1.291s
2022-05-18T04:32:20.1934725Z 
2022-05-18T04:32:20.1934797Z OK (skipped=1)
2022-05-18T04:32:20.1934893Z 
2022-05-18T04:32:20.1934978Z Generating XML reports...
2022-05-18T04:32:20.1964768Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043218.xml
2022-05-18T04:32:21.1090539Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:32:21.1100465Z 
2022-05-18T04:32:21.1100568Z Running tests...
2022-05-18T04:32:21.1101299Z ----------------------------------------------------------------------
2022-05-18T04:32:21.3908762Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27848
2022-05-18T04:32:21.3930640Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27849
2022-05-18T04:32:21.3953687Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27850
2022-05-18T04:32:22.1968685Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:32:22.2071417Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:32:22.2071846Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:32:22.2072520Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:32:22.2073052Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:32:22.2073573Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:32:22.2080228Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:32:22.2081116Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:32:22.2081769Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:32:22.4002578Z skip: Need at least 2 CUDA devices (1.290s)
2022-05-18T04:32:22.4002905Z 
2022-05-18T04:32:22.4003425Z ----------------------------------------------------------------------
2022-05-18T04:32:22.4003726Z Ran 1 test in 1.290s
2022-05-18T04:32:22.4003843Z 
2022-05-18T04:32:22.4003906Z OK (skipped=1)
2022-05-18T04:32:22.4004014Z 
2022-05-18T04:32:22.4004103Z Generating XML reports...
2022-05-18T04:32:22.4034254Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043221.xml
2022-05-18T04:32:23.3235585Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:32:23.3245075Z 
2022-05-18T04:32:23.3245523Z Running tests...
2022-05-18T04:32:23.3246122Z ----------------------------------------------------------------------
2022-05-18T04:32:23.6077780Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27901
2022-05-18T04:32:23.6101336Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27902
2022-05-18T04:32:23.6124284Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27903
2022-05-18T04:32:24.4006788Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:32:24.4108113Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:32:24.4108540Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:32:24.4109204Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:32:24.4110071Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:32:24.4110930Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:32:24.4118565Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:32:24.4119177Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:32:24.4120252Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:32:24.6173934Z skip: Need at least 2 CUDA devices (1.293s)
2022-05-18T04:32:24.6174232Z 
2022-05-18T04:32:24.6174750Z ----------------------------------------------------------------------
2022-05-18T04:32:24.6175079Z Ran 1 test in 1.293s
2022-05-18T04:32:24.6175194Z 
2022-05-18T04:32:24.6175267Z OK (skipped=1)
2022-05-18T04:32:24.6175375Z 
2022-05-18T04:32:24.6175461Z Generating XML reports...
2022-05-18T04:32:24.6205873Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043223.xml
2022-05-18T04:32:25.5411259Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:32:25.5421012Z 
2022-05-18T04:32:25.5421128Z Running tests...
2022-05-18T04:32:25.5421744Z ----------------------------------------------------------------------
2022-05-18T04:32:25.8232742Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27954
2022-05-18T04:32:25.8253406Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27955
2022-05-18T04:32:25.8276251Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27956
2022-05-18T04:32:26.6300395Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:32:26.6401945Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:32:26.6402757Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:32:26.6403827Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:32:26.6404683Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:32:26.6405191Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:32:26.6411282Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:32:26.6413675Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:32:26.6414072Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:32:26.8325486Z skip: Need at least 2 CUDA devices (1.290s)
2022-05-18T04:32:26.8325770Z 
2022-05-18T04:32:26.8326277Z ----------------------------------------------------------------------
2022-05-18T04:32:26.8326729Z Ran 1 test in 1.290s
2022-05-18T04:32:26.8326945Z 
2022-05-18T04:32:26.8327037Z OK (skipped=1)
2022-05-18T04:32:26.8327144Z 
2022-05-18T04:32:26.8327215Z Generating XML reports...
2022-05-18T04:32:26.8357367Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043225.xml
2022-05-18T04:32:27.7696044Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:32:27.7706911Z 
2022-05-18T04:32:27.7707275Z Running tests...
2022-05-18T04:32:27.7707690Z ----------------------------------------------------------------------
2022-05-18T04:32:28.0506295Z   test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28007
2022-05-18T04:32:28.0528825Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28008
2022-05-18T04:32:28.0552409Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28009
2022-05-18T04:32:28.8740496Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:32:28.8842242Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:32:28.8842883Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:32:28.8843524Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:32:28.8844269Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:32:28.8845110Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:32:28.8853773Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:32:28.8854453Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:32:28.8855087Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:32:29.0603283Z skip: Need at least 2 CUDA devices (1.289s)
2022-05-18T04:32:29.0603594Z 
2022-05-18T04:32:29.0604048Z ----------------------------------------------------------------------
2022-05-18T04:32:29.0604471Z Ran 1 test in 1.290s
2022-05-18T04:32:29.0604641Z 
2022-05-18T04:32:29.0604767Z OK (skipped=1)
2022-05-18T04:32:29.0604944Z 
2022-05-18T04:32:29.0605080Z Generating XML reports...
2022-05-18T04:32:29.0636411Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043227.xml
2022-05-18T04:32:29.9872483Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:32:29.9883409Z 
2022-05-18T04:32:29.9883833Z Running tests...
2022-05-18T04:32:29.9884208Z ----------------------------------------------------------------------
2022-05-18T04:32:30.2698265Z   test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28060
2022-05-18T04:32:30.2720170Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28061
2022-05-18T04:32:30.2743587Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28062
2022-05-18T04:32:31.0918387Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:32:31.0918869Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:32:31.0919242Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:32:31.0919863Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:32:31.0920378Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:32:31.0920901Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:32:31.1024538Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:32:31.1929990Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:32:31.1930577Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:32:31.3794294Z skip: Need at least 2 CUDA devices (1.391s)
2022-05-18T04:32:31.3794669Z 
2022-05-18T04:32:31.3795128Z ----------------------------------------------------------------------
2022-05-18T04:32:31.3795376Z Ran 1 test in 1.391s
2022-05-18T04:32:31.3795493Z 
2022-05-18T04:32:31.3795567Z OK (skipped=1)
2022-05-18T04:32:31.3795661Z 
2022-05-18T04:32:31.3795750Z Generating XML reports...
2022-05-18T04:32:31.3826165Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043229.xml
2022-05-18T04:32:32.3544415Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:32:32.3554301Z 
2022-05-18T04:32:32.3554442Z Running tests...
2022-05-18T04:32:32.3555034Z ----------------------------------------------------------------------
2022-05-18T04:32:32.3569696Z   test_ddp_ignore_params_arg (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'nccl'} (0.001s)
2022-05-18T04:32:32.3570098Z 
2022-05-18T04:32:32.3570479Z ----------------------------------------------------------------------
2022-05-18T04:32:32.3570844Z Ran 1 test in 0.002s
2022-05-18T04:32:32.3570959Z 
2022-05-18T04:32:32.3571034Z OK (skipped=1)
2022-05-18T04:32:32.3571141Z 
2022-05-18T04:32:32.3571227Z Generating XML reports...
2022-05-18T04:32:32.3602274Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043232.xml
2022-05-18T04:32:33.2125318Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:32:33.2134957Z 
2022-05-18T04:32:33.2135092Z Running tests...
2022-05-18T04:32:33.2135971Z ----------------------------------------------------------------------
2022-05-18T04:32:33.5030659Z   test_ddp_inference (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28123
2022-05-18T04:32:33.5052870Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28124
2022-05-18T04:32:33.5075332Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28125
2022-05-18T04:32:34.3309642Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:32:34.3310358Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:32:34.3310868Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:32:34.3311516Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:32:34.3312289Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:32:34.3312868Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:32:34.3416365Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:32:34.4325347Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:32:34.4326044Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:32:34.6126661Z skip: Need at least 2 CUDA devices (1.399s)
2022-05-18T04:32:34.6126950Z 
2022-05-18T04:32:34.6127482Z ----------------------------------------------------------------------
2022-05-18T04:32:34.6127849Z Ran 1 test in 1.399s
2022-05-18T04:32:34.6127965Z 
2022-05-18T04:32:34.6128043Z OK (skipped=1)
2022-05-18T04:32:34.6128160Z 
2022-05-18T04:32:34.6128249Z Generating XML reports...
2022-05-18T04:32:34.6157613Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043233.xml
2022-05-18T04:32:35.5333970Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:32:35.5343475Z 
2022-05-18T04:32:35.5343619Z Running tests...
2022-05-18T04:32:35.5344231Z ----------------------------------------------------------------------
2022-05-18T04:32:35.8179701Z   test_ddp_join_model_equivalence (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28176
2022-05-18T04:32:35.8201879Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28177
2022-05-18T04:32:35.8224431Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28178
2022-05-18T04:32:36.6429661Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:32:36.6530883Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:32:36.6531636Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:32:36.6532352Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:32:36.6532877Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:32:36.6533405Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:32:36.6638448Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:32:36.7545966Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:32:36.7546387Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:32:36.9273462Z skip: Need at least 2 CUDA devices (1.393s)
2022-05-18T04:32:36.9273812Z 
2022-05-18T04:32:36.9274353Z ----------------------------------------------------------------------
2022-05-18T04:32:36.9274800Z Ran 1 test in 1.393s
2022-05-18T04:32:36.9275009Z 
2022-05-18T04:32:36.9275139Z OK (skipped=1)
2022-05-18T04:32:36.9275323Z 
2022-05-18T04:32:36.9275485Z Generating XML reports...
2022-05-18T04:32:36.9305649Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043235.xml
2022-05-18T04:32:37.8524799Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:32:37.8534984Z 
2022-05-18T04:32:37.8535089Z Running tests...
2022-05-18T04:32:37.8535498Z ----------------------------------------------------------------------
2022-05-18T04:32:38.1371575Z   test_ddp_logging_data_cpu (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28229
2022-05-18T04:32:38.1394064Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28230
2022-05-18T04:32:38.1416960Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28231
2022-05-18T04:32:38.9527468Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:32:38.9628944Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:32:38.9629814Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:32:38.9630233Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:32:38.9630908Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:32:38.9631781Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:32:38.9639574Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:32:38.9640874Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:32:38.9641344Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:32:38.9709848Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplv547emy
2022-05-18T04:32:38.9711500Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplv547emy/_remote_module_non_scriptable.py
2022-05-18T04:32:38.9712417Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5knddmwf
2022-05-18T04:32:38.9712913Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvvc1xlzj
2022-05-18T04:32:38.9714019Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5knddmwf/_remote_module_non_scriptable.py
2022-05-18T04:32:38.9714434Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvvc1xlzj/_remote_module_non_scriptable.py
2022-05-18T04:32:38.9852017Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:32:38.9852449Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:32:38.9852813Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-05-18T04:32:39.3469177Z ok (1.493s)
2022-05-18T04:32:39.3469418Z 
2022-05-18T04:32:39.3469970Z ----------------------------------------------------------------------
2022-05-18T04:32:39.3470257Z Ran 1 test in 1.493s
2022-05-18T04:32:39.3470374Z 
2022-05-18T04:32:39.3470438Z OK
2022-05-18T04:32:39.3470531Z 
2022-05-18T04:32:39.3470624Z Generating XML reports...
2022-05-18T04:32:39.3501359Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043237.xml
2022-05-18T04:32:40.2694354Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:32:40.2704282Z 
2022-05-18T04:32:40.2704415Z Running tests...
2022-05-18T04:32:40.2705020Z ----------------------------------------------------------------------
2022-05-18T04:32:40.5513766Z   test_ddp_logging_data_gpu (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28291
2022-05-18T04:32:40.5535885Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28292
2022-05-18T04:32:40.5559167Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28293
2022-05-18T04:32:41.3670376Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:32:41.3771491Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:32:41.3772357Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:32:41.3773457Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:32:41.3774271Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:32:41.3774802Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:32:41.3781858Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:32:41.3782334Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:32:41.3784029Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:32:41.5608370Z skip: CUDA is not available. (1.290s)
2022-05-18T04:32:41.5608694Z 
2022-05-18T04:32:41.5609228Z ----------------------------------------------------------------------
2022-05-18T04:32:41.5609513Z Ran 1 test in 1.290s
2022-05-18T04:32:41.5609637Z 
2022-05-18T04:32:41.5609712Z OK (skipped=1)
2022-05-18T04:32:41.5609821Z 
2022-05-18T04:32:41.5610895Z Generating XML reports...
2022-05-18T04:32:41.5643511Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043240.xml
2022-05-18T04:32:42.4742833Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:32:42.4753771Z 
2022-05-18T04:32:42.4754170Z Running tests...
2022-05-18T04:32:42.4754601Z ----------------------------------------------------------------------
2022-05-18T04:32:42.4774538Z   test_ddp_model_diff_num_params_across_ranks (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'nccl'} (0.002s)
2022-05-18T04:32:42.4774846Z 
2022-05-18T04:32:42.4775067Z ----------------------------------------------------------------------
2022-05-18T04:32:42.4775299Z Ran 1 test in 0.002s
2022-05-18T04:32:42.4775415Z 
2022-05-18T04:32:42.4775493Z OK (skipped=1)
2022-05-18T04:32:42.4775601Z 
2022-05-18T04:32:42.4775687Z Generating XML reports...
2022-05-18T04:32:42.4806200Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043242.xml
2022-05-18T04:32:43.3104866Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:32:43.3114896Z 
2022-05-18T04:32:43.3115170Z Running tests...
2022-05-18T04:32:43.3115791Z ----------------------------------------------------------------------
2022-05-18T04:32:43.3134545Z   test_ddp_model_diff_shape_across_ranks (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'nccl', 'gloo'} (0.002s)
2022-05-18T04:32:43.3134970Z 
2022-05-18T04:32:43.3135394Z ----------------------------------------------------------------------
2022-05-18T04:32:43.3135788Z Ran 1 test in 0.002s
2022-05-18T04:32:43.3135904Z 
2022-05-18T04:32:43.3135978Z OK (skipped=1)
2022-05-18T04:32:43.3136088Z 
2022-05-18T04:32:43.3136182Z Generating XML reports...
2022-05-18T04:32:43.3166368Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043243.xml
2022-05-18T04:32:44.1464850Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:32:44.1474709Z 
2022-05-18T04:32:44.1475008Z Running tests...
2022-05-18T04:32:44.1475675Z ----------------------------------------------------------------------
2022-05-18T04:32:44.1490351Z   test_ddp_multiple_nested_unused_params_err_ignore_params (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'nccl'} (0.001s)
2022-05-18T04:32:44.1490711Z 
2022-05-18T04:32:44.1491064Z ----------------------------------------------------------------------
2022-05-18T04:32:44.1491801Z Ran 1 test in 0.002s
2022-05-18T04:32:44.1491915Z 
2022-05-18T04:32:44.1491975Z OK (skipped=1)
2022-05-18T04:32:44.1492085Z 
2022-05-18T04:32:44.1492238Z Generating XML reports...
2022-05-18T04:32:44.1523655Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043244.xml
2022-05-18T04:32:44.9827485Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:32:44.9838627Z 
2022-05-18T04:32:44.9839083Z Running tests...
2022-05-18T04:32:44.9839501Z ----------------------------------------------------------------------
2022-05-18T04:32:44.9854950Z   test_ddp_multiple_nested_unused_params_error (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'nccl', 'gloo'} (0.001s)
2022-05-18T04:32:44.9855439Z 
2022-05-18T04:32:44.9855793Z ----------------------------------------------------------------------
2022-05-18T04:32:44.9856218Z Ran 1 test in 0.002s
2022-05-18T04:32:44.9856407Z 
2022-05-18T04:32:44.9856529Z OK (skipped=1)
2022-05-18T04:32:44.9856707Z 
2022-05-18T04:32:44.9856831Z Generating XML reports...
2022-05-18T04:32:44.9901215Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043244.xml
2022-05-18T04:32:45.8193074Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:32:45.8203695Z 
2022-05-18T04:32:45.8204003Z Running tests...
2022-05-18T04:32:45.8204622Z ----------------------------------------------------------------------
2022-05-18T04:32:45.8228789Z   test_ddp_namedtuple (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'nccl'} (0.002s)
2022-05-18T04:32:45.8229217Z 
2022-05-18T04:32:45.8229572Z ----------------------------------------------------------------------
2022-05-18T04:32:45.8229971Z Ran 1 test in 0.002s
2022-05-18T04:32:45.8230153Z 
2022-05-18T04:32:45.8230292Z OK (skipped=1)
2022-05-18T04:32:45.8230451Z 
2022-05-18T04:32:45.8230592Z Generating XML reports...
2022-05-18T04:32:45.8261631Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043245.xml
2022-05-18T04:32:46.6589912Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:32:46.6600360Z 
2022-05-18T04:32:46.6600848Z Running tests...
2022-05-18T04:32:46.6601478Z ----------------------------------------------------------------------
2022-05-18T04:32:46.9413056Z   test_ddp_new_tensor_in_fwd (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28394
2022-05-18T04:32:46.9436109Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28395
2022-05-18T04:32:46.9458884Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28396
2022-05-18T04:32:47.8527812Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:32:47.8628910Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:32:47.8629398Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:32:47.8630019Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:32:47.8630558Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:32:47.8631421Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:32:47.8639602Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:32:47.8640803Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:32:47.8641431Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:32:48.0509903Z skip: Need at least 2 CUDA devices (1.391s)
2022-05-18T04:32:48.0510349Z 
2022-05-18T04:32:48.0511467Z ----------------------------------------------------------------------
2022-05-18T04:32:48.0511817Z Ran 1 test in 1.391s
2022-05-18T04:32:48.0511999Z 
2022-05-18T04:32:48.0512081Z OK (skipped=1)
2022-05-18T04:32:48.0512205Z 
2022-05-18T04:32:48.0512292Z Generating XML reports...
2022-05-18T04:32:48.0541764Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043246.xml
2022-05-18T04:32:48.9703562Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:32:48.9713454Z 
2022-05-18T04:32:48.9713780Z Running tests...
2022-05-18T04:32:48.9714427Z ----------------------------------------------------------------------
2022-05-18T04:32:49.2511532Z   test_ddp_new_tensor_in_fwd_static_graph (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28447
2022-05-18T04:32:49.2533974Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28448
2022-05-18T04:32:49.2556962Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28449
2022-05-18T04:32:50.0722855Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:32:50.0795626Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:32:50.0796113Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:32:50.0796748Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:32:50.0797276Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:32:50.0824213Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:32:50.0904495Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:32:50.0905774Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:32:50.1837515Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:32:50.3610672Z skip: Need at least 2 CUDA devices (1.389s)
2022-05-18T04:32:50.3610948Z 
2022-05-18T04:32:50.3611413Z ----------------------------------------------------------------------
2022-05-18T04:32:50.3611874Z Ran 1 test in 1.390s
2022-05-18T04:32:50.3611994Z 
2022-05-18T04:32:50.3612066Z OK (skipped=1)
2022-05-18T04:32:50.3612173Z 
2022-05-18T04:32:50.3612258Z Generating XML reports...
2022-05-18T04:32:50.3642953Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043248.xml
2022-05-18T04:32:51.2785323Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:32:51.2795369Z 
2022-05-18T04:32:51.2795504Z Running tests...
2022-05-18T04:32:51.2796113Z ----------------------------------------------------------------------
2022-05-18T04:32:51.2811711Z   test_ddp_profiling_autograd_profiler (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'nccl'} (0.001s)
2022-05-18T04:32:51.2812222Z 
2022-05-18T04:32:51.2812592Z ----------------------------------------------------------------------
2022-05-18T04:32:51.2812843Z Ran 1 test in 0.002s
2022-05-18T04:32:51.2812960Z 
2022-05-18T04:32:51.2813034Z OK (skipped=1)
2022-05-18T04:32:51.2813128Z 
2022-05-18T04:32:51.2813214Z Generating XML reports...
2022-05-18T04:32:51.2843545Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043251.xml
2022-05-18T04:32:52.1030850Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:32:52.1041311Z 
2022-05-18T04:32:52.1041653Z Running tests...
2022-05-18T04:32:52.1041997Z ----------------------------------------------------------------------
2022-05-18T04:32:52.1060126Z   test_ddp_profiling_torch_profiler (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'nccl', 'gloo'} (0.002s)
2022-05-18T04:32:52.1060584Z 
2022-05-18T04:32:52.1060940Z ----------------------------------------------------------------------
2022-05-18T04:32:52.1061327Z Ran 1 test in 0.002s
2022-05-18T04:32:52.1061516Z 
2022-05-18T04:32:52.1061638Z OK (skipped=1)
2022-05-18T04:32:52.1061823Z 
2022-05-18T04:32:52.1061967Z Generating XML reports...
2022-05-18T04:32:52.1092979Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043252.xml
2022-05-18T04:32:52.9440194Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:32:52.9449793Z 
2022-05-18T04:32:52.9449993Z Running tests...
2022-05-18T04:32:52.9450341Z ----------------------------------------------------------------------
2022-05-18T04:32:53.2260428Z   test_ddp_python_error_logged (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28520
2022-05-18T04:32:53.2283719Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28521
2022-05-18T04:32:53.2306155Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28522
2022-05-18T04:32:54.0355834Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:32:54.0457790Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:32:54.0458210Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:32:54.0458856Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:32:54.0459400Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:32:54.0459926Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:32:54.0468095Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:32:54.0469044Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:32:54.0469585Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:32:54.2356817Z skip: Need at least 2 CUDA devices (1.290s)
2022-05-18T04:32:54.2357079Z 
2022-05-18T04:32:54.2357544Z ----------------------------------------------------------------------
2022-05-18T04:32:54.2357958Z Ran 1 test in 1.291s
2022-05-18T04:32:54.2358137Z 
2022-05-18T04:32:54.2358253Z OK (skipped=1)
2022-05-18T04:32:54.2358429Z 
2022-05-18T04:32:54.2358565Z Generating XML reports...
2022-05-18T04:32:54.2397750Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043252.xml
2022-05-18T04:32:55.1557283Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:32:55.1566770Z 
2022-05-18T04:32:55.1566904Z Running tests...
2022-05-18T04:32:55.1567318Z ----------------------------------------------------------------------
2022-05-18T04:32:55.4422068Z   test_ddp_returns_tensor_with_no_grad (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28573
2022-05-18T04:32:55.4444403Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28574
2022-05-18T04:32:55.4467008Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28575
2022-05-18T04:32:56.2464943Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:32:56.2565542Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:32:56.2565949Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:32:56.2566555Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:32:56.2567092Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:32:56.2567619Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:32:56.2673012Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:32:56.3580269Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:32:56.3580765Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:32:56.5520854Z skip: Need at least 2 CUDA devices (1.395s)
2022-05-18T04:32:56.5521099Z 
2022-05-18T04:32:56.5521647Z ----------------------------------------------------------------------
2022-05-18T04:32:56.5522073Z Ran 1 test in 1.395s
2022-05-18T04:32:56.5522286Z 
2022-05-18T04:32:56.5522422Z OK (skipped=1)
2022-05-18T04:32:56.5522549Z 
2022-05-18T04:32:56.5522640Z Generating XML reports...
2022-05-18T04:32:56.5552941Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043255.xml
2022-05-18T04:32:57.4787696Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:32:57.4798008Z 
2022-05-18T04:32:57.4798159Z Running tests...
2022-05-18T04:32:57.4799389Z ----------------------------------------------------------------------
2022-05-18T04:32:57.4822353Z   test_ddp_shared_grad_acc_unused_params (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'nccl'} (0.002s)
2022-05-18T04:32:57.4822798Z 
2022-05-18T04:32:57.4823210Z ----------------------------------------------------------------------
2022-05-18T04:32:57.4823459Z Ran 1 test in 0.002s
2022-05-18T04:32:57.4823575Z 
2022-05-18T04:32:57.4823637Z OK (skipped=1)
2022-05-18T04:32:57.4823745Z 
2022-05-18T04:32:57.4823831Z Generating XML reports...
2022-05-18T04:32:57.4854501Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043257.xml
2022-05-18T04:32:58.3128352Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:32:58.3138064Z 
2022-05-18T04:32:58.3138179Z Running tests...
2022-05-18T04:32:58.3138801Z ----------------------------------------------------------------------
2022-05-18T04:32:58.5894199Z   test_ddp_static_graph_nested_types (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77625 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure IN_CI is not set and you are not using the flag --import-disabled-tests. (0.275s)
2022-05-18T04:32:58.5894716Z 
2022-05-18T04:32:58.5894925Z ----------------------------------------------------------------------
2022-05-18T04:32:58.5895173Z Ran 1 test in 0.275s
2022-05-18T04:32:58.5895287Z 
2022-05-18T04:32:58.5895348Z OK (skipped=1)
2022-05-18T04:32:58.5895456Z 
2022-05-18T04:32:58.5895543Z Generating XML reports...
2022-05-18T04:32:58.5922373Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043258.xml
2022-05-18T04:32:59.4787313Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:32:59.4797187Z 
2022-05-18T04:32:59.4797286Z Running tests...
2022-05-18T04:32:59.4798450Z ----------------------------------------------------------------------
2022-05-18T04:32:59.7606126Z   test_ddp_sync_bn_training_vs_eval (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28646
2022-05-18T04:32:59.7628740Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28647
2022-05-18T04:32:59.7652808Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28648
2022-05-18T04:33:00.5485723Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:33:00.5486307Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:33:00.5486682Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:33:00.5487313Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:33:00.5487851Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:33:00.5488374Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:33:00.5495951Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:33:00.5496574Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:33:00.5498496Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:33:00.7702256Z skip: Need at least 2 CUDA devices (1.290s)
2022-05-18T04:33:00.7702522Z 
2022-05-18T04:33:00.7703098Z ----------------------------------------------------------------------
2022-05-18T04:33:00.7703511Z Ran 1 test in 1.290s
2022-05-18T04:33:00.7703689Z 
2022-05-18T04:33:00.7703801Z OK (skipped=1)
2022-05-18T04:33:00.7703976Z 
2022-05-18T04:33:00.7704106Z Generating XML reports...
2022-05-18T04:33:00.7735981Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043259.xml
2022-05-18T04:33:01.7010354Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:33:01.7020101Z 
2022-05-18T04:33:01.7020183Z Running tests...
2022-05-18T04:33:01.7020604Z ----------------------------------------------------------------------
2022-05-18T04:33:01.9900025Z   test_ddp_sync_module_states (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28699
2022-05-18T04:33:01.9923295Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28700
2022-05-18T04:33:01.9945950Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28701
2022-05-18T04:33:02.8159827Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:33:02.8261113Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:33:02.8261555Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:33:02.8262173Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:33:02.8262695Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:33:02.8263346Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:33:02.8368311Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:33:02.9276234Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:33:02.9276623Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:33:03.0996358Z skip: Need at least 2 CUDA devices (1.397s)
2022-05-18T04:33:03.0996670Z 
2022-05-18T04:33:03.0996998Z ----------------------------------------------------------------------
2022-05-18T04:33:03.0997252Z Ran 1 test in 1.397s
2022-05-18T04:33:03.0997353Z 
2022-05-18T04:33:03.0997428Z OK (skipped=1)
2022-05-18T04:33:03.0997537Z 
2022-05-18T04:33:03.0997624Z Generating XML reports...
2022-05-18T04:33:03.1027339Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043301.xml
2022-05-18T04:33:04.0204065Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:33:04.0212998Z 
2022-05-18T04:33:04.0213117Z Running tests...
2022-05-18T04:33:04.0213730Z ----------------------------------------------------------------------
2022-05-18T04:33:04.3031967Z   test_ddp_uneven_input_exception (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28752
2022-05-18T04:33:04.3054976Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28753
2022-05-18T04:33:04.3078320Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28754
2022-05-18T04:33:05.1199506Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:33:05.1299960Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:33:05.1300686Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:33:05.1301387Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:33:05.1301921Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:33:05.1302469Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:33:05.1408322Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:33:05.2316110Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:33:05.2316848Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:33:05.4130837Z skip: Need at least 2 CUDA devices (1.391s)
2022-05-18T04:33:05.4131157Z 
2022-05-18T04:33:05.4131567Z ----------------------------------------------------------------------
2022-05-18T04:33:05.4131994Z Ran 1 test in 1.392s
2022-05-18T04:33:05.4132190Z 
2022-05-18T04:33:05.4132335Z OK (skipped=1)
2022-05-18T04:33:05.4132540Z 
2022-05-18T04:33:05.4132676Z Generating XML reports...
2022-05-18T04:33:05.4163809Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043304.xml
2022-05-18T04:33:06.3465271Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:33:06.3475175Z 
2022-05-18T04:33:06.3475322Z Running tests...
2022-05-18T04:33:06.3475939Z ----------------------------------------------------------------------
2022-05-18T04:33:06.6285879Z   test_ddp_uneven_input_join_disable (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28805
2022-05-18T04:33:06.6308645Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28806
2022-05-18T04:33:06.6331721Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28807
2022-05-18T04:33:07.4587240Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:33:07.4688146Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:33:07.4688874Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:33:07.4689851Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:33:07.4690637Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:33:07.4691515Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:33:07.4799173Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:33:07.4799672Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:33:07.5703125Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:33:07.7383105Z skip: Need at least 2 CUDA devices (1.390s)
2022-05-18T04:33:07.7383273Z 
2022-05-18T04:33:07.7383706Z ----------------------------------------------------------------------
2022-05-18T04:33:07.7384142Z Ran 1 test in 1.391s
2022-05-18T04:33:07.7384352Z 
2022-05-18T04:33:07.7384497Z OK (skipped=1)
2022-05-18T04:33:07.7384631Z 
2022-05-18T04:33:07.7384707Z Generating XML reports...
2022-05-18T04:33:07.7415091Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043306.xml
2022-05-18T04:33:08.6735169Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:33:08.6745208Z 
2022-05-18T04:33:08.6745304Z Running tests...
2022-05-18T04:33:08.6745936Z ----------------------------------------------------------------------
2022-05-18T04:33:08.9510545Z   test_ddp_uneven_inputs (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/75648 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure IN_CI is not set and you are not using the flag --import-disabled-tests. (0.276s)
2022-05-18T04:33:08.9511081Z 
2022-05-18T04:33:08.9511307Z ----------------------------------------------------------------------
2022-05-18T04:33:08.9511541Z Ran 1 test in 0.276s
2022-05-18T04:33:08.9511717Z 
2022-05-18T04:33:08.9511796Z OK (skipped=1)
2022-05-18T04:33:08.9511905Z 
2022-05-18T04:33:08.9511993Z Generating XML reports...
2022-05-18T04:33:08.9539019Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043308.xml
2022-05-18T04:33:09.8432529Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:33:09.8442519Z 
2022-05-18T04:33:09.8442710Z Running tests...
2022-05-18T04:33:09.8443315Z ----------------------------------------------------------------------
2022-05-18T04:33:10.1287225Z   test_ddp_uneven_inputs_stop_iteration_sync_bn (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28868
2022-05-18T04:33:10.1309395Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28869
2022-05-18T04:33:10.1332049Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28870
2022-05-18T04:33:10.9394527Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:33:10.9395250Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:33:10.9395631Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:33:10.9396240Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:33:10.9396933Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:33:10.9397504Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:33:10.9501827Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:33:10.9502325Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:33:11.0407313Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:33:11.2382699Z skip: Need at least 2 CUDA devices (1.394s)
2022-05-18T04:33:11.2383083Z 
2022-05-18T04:33:11.2383494Z ----------------------------------------------------------------------
2022-05-18T04:33:11.2383758Z Ran 1 test in 1.394s
2022-05-18T04:33:11.2383876Z 
2022-05-18T04:33:11.2383967Z OK (skipped=1)
2022-05-18T04:33:11.2384114Z 
2022-05-18T04:33:11.2384226Z Generating XML reports...
2022-05-18T04:33:11.2414597Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043309.xml
2022-05-18T04:33:12.1584881Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:33:12.1595039Z 
2022-05-18T04:33:12.1595178Z Running tests...
2022-05-18T04:33:12.1595788Z ----------------------------------------------------------------------
2022-05-18T04:33:12.1622267Z   test_ddp_unused_params_rebuild_buckets_exception (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'nccl', 'gloo'} (0.003s)
2022-05-18T04:33:12.1622786Z 
2022-05-18T04:33:12.1623232Z ----------------------------------------------------------------------
2022-05-18T04:33:12.1623482Z Ran 1 test in 0.003s
2022-05-18T04:33:12.1623601Z 
2022-05-18T04:33:12.1623663Z OK (skipped=1)
2022-05-18T04:33:12.1623772Z 
2022-05-18T04:33:12.1623862Z Generating XML reports...
2022-05-18T04:33:12.1654695Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043312.xml
2022-05-18T04:33:12.9949507Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:33:12.9959640Z 
2022-05-18T04:33:12.9959836Z Running tests...
2022-05-18T04:33:12.9960474Z ----------------------------------------------------------------------
2022-05-18T04:33:13.2802061Z   test_destroy_full_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28931
2022-05-18T04:33:13.2824519Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28932
2022-05-18T04:33:13.2846663Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28933
2022-05-18T04:33:14.0844574Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:33:14.0945611Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:33:14.0946125Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:33:14.0946732Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:33:14.0947262Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:33:14.0947777Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:33:14.0955359Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:33:14.0956064Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:33:14.0956852Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:33:14.1064729Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:33:14.1167242Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-05-18T04:33:14.1167996Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:33:14.1168618Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:33:14.1169145Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:33:14.1267793Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:33:14.3898297Z ok (1.394s)
2022-05-18T04:33:14.3898556Z 
2022-05-18T04:33:14.3899084Z ----------------------------------------------------------------------
2022-05-18T04:33:14.3899518Z Ran 1 test in 1.394s
2022-05-18T04:33:14.3899669Z 
2022-05-18T04:33:14.3899738Z OK
2022-05-18T04:33:14.3899833Z 
2022-05-18T04:33:14.3899927Z Generating XML reports...
2022-05-18T04:33:14.3938426Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043312.xml
2022-05-18T04:33:15.3152460Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:33:15.3163285Z 
2022-05-18T04:33:15.3163389Z Running tests...
2022-05-18T04:33:15.3164084Z ----------------------------------------------------------------------
2022-05-18T04:33:15.6007888Z   test_destroy_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28993
2022-05-18T04:33:15.6031096Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28994
2022-05-18T04:33:15.6054142Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28995
2022-05-18T04:33:16.3972223Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:33:16.3972939Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:33:16.3973525Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:33:16.3974258Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:33:16.3974957Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:33:16.3975836Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:33:16.4078721Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:33:16.4985499Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:33:16.4986145Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:33:16.4987136Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:33:16.5190425Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:33:16.5191032Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-05-18T04:33:16.5191840Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:33:16.5192381Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:33:16.5192893Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:33:16.9108180Z ok (1.594s)
2022-05-18T04:33:16.9108420Z 
2022-05-18T04:33:16.9108943Z ----------------------------------------------------------------------
2022-05-18T04:33:16.9109315Z Ran 1 test in 1.594s
2022-05-18T04:33:16.9109617Z 
2022-05-18T04:33:16.9109682Z OK
2022-05-18T04:33:16.9109778Z 
2022-05-18T04:33:16.9109875Z Generating XML reports...
2022-05-18T04:33:16.9142638Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043315.xml
2022-05-18T04:33:17.8421879Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:33:17.8431717Z 
2022-05-18T04:33:17.8431861Z Running tests...
2022-05-18T04:33:17.8432469Z ----------------------------------------------------------------------
2022-05-18T04:33:18.1251092Z   test_detect_ddp_is_actually_static (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29052
2022-05-18T04:33:18.1272942Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29053
2022-05-18T04:33:18.1295907Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29054
2022-05-18T04:33:18.9439509Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:33:18.9540744Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:33:18.9541369Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:33:18.9541997Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:33:18.9542534Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:33:18.9543142Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:33:18.9551689Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:33:18.9552180Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:33:18.9553374Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:33:19.1344526Z skip: Need at least 2 CUDA devices (1.291s)
2022-05-18T04:33:19.1344722Z 
2022-05-18T04:33:19.1345032Z ----------------------------------------------------------------------
2022-05-18T04:33:19.1345267Z Ran 1 test in 1.291s
2022-05-18T04:33:19.1345382Z 
2022-05-18T04:33:19.1345456Z OK (skipped=1)
2022-05-18T04:33:19.1345564Z 
2022-05-18T04:33:19.1345649Z Generating XML reports...
2022-05-18T04:33:19.1376029Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043317.xml
2022-05-18T04:33:20.0579127Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:33:20.0589237Z 
2022-05-18T04:33:20.0589460Z Running tests...
2022-05-18T04:33:20.0589870Z ----------------------------------------------------------------------
2022-05-18T04:33:20.0607165Z   test_different_graph_across_ranks (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'nccl', 'gloo'} (0.002s)
2022-05-18T04:33:20.0607494Z 
2022-05-18T04:33:20.0607696Z ----------------------------------------------------------------------
2022-05-18T04:33:20.0607961Z Ran 1 test in 0.002s
2022-05-18T04:33:20.0608134Z 
2022-05-18T04:33:20.0608223Z OK (skipped=1)
2022-05-18T04:33:20.0608333Z 
2022-05-18T04:33:20.0608419Z Generating XML reports...
2022-05-18T04:33:20.0639351Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043320.xml
2022-05-18T04:33:20.8946234Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:33:20.8956201Z 
2022-05-18T04:33:20.8956455Z Running tests...
2022-05-18T04:33:20.8957099Z ----------------------------------------------------------------------
2022-05-18T04:33:21.1765600Z   test_dump_DDP_relevant_env_vars (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29115
2022-05-18T04:33:21.1787771Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29116
2022-05-18T04:33:21.1810863Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29117
2022-05-18T04:33:21.9803886Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:33:21.9876512Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:33:21.9877214Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:33:21.9877867Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:33:21.9878407Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:33:21.9905268Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:33:21.9985962Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:33:21.9986498Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:33:22.0917765Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:33:22.2864111Z ok (1.390s)
2022-05-18T04:33:22.2864342Z 
2022-05-18T04:33:22.2864878Z ----------------------------------------------------------------------
2022-05-18T04:33:22.2865293Z Ran 1 test in 1.391s
2022-05-18T04:33:22.2865409Z 
2022-05-18T04:33:22.2865478Z OK
2022-05-18T04:33:22.2865557Z 
2022-05-18T04:33:22.2865650Z Generating XML reports...
2022-05-18T04:33:22.2905126Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043320.xml
2022-05-18T04:33:23.2172858Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:33:23.2182600Z 
2022-05-18T04:33:23.2182832Z Running tests...
2022-05-18T04:33:23.2183323Z ----------------------------------------------------------------------
2022-05-18T04:33:23.5088976Z   test_gather (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29168
2022-05-18T04:33:23.5113330Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29169
2022-05-18T04:33:23.5136412Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29170
2022-05-18T04:33:24.3611701Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:33:24.3712472Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:33:24.3713049Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:33:24.3714009Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:33:24.3714871Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:33:24.3715726Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:33:24.3820880Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:33:24.4727984Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:33:24.4728539Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:33:24.8191201Z ok (1.601s)
2022-05-18T04:33:24.8191593Z 
2022-05-18T04:33:24.8192264Z ----------------------------------------------------------------------
2022-05-18T04:33:24.8192537Z Ran 1 test in 1.601s
2022-05-18T04:33:24.8192653Z 
2022-05-18T04:33:24.8192703Z OK
2022-05-18T04:33:24.8192797Z 
2022-05-18T04:33:24.8192889Z Generating XML reports...
2022-05-18T04:33:24.8223187Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043323.xml
2022-05-18T04:33:25.7414615Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:33:25.7425501Z 
2022-05-18T04:33:25.7425916Z Running tests...
2022-05-18T04:33:25.7426389Z ----------------------------------------------------------------------
2022-05-18T04:33:26.0241123Z   test_gather_checks (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29224
2022-05-18T04:33:26.0262829Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29225
2022-05-18T04:33:26.0285527Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29226
2022-05-18T04:33:26.8571101Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:33:26.8672224Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:33:26.8672895Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:33:26.8673844Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:33:26.8674499Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:33:26.8675007Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:33:26.8682315Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:33:26.8683263Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:33:26.8684162Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:33:27.1336764Z ok (1.391s)
2022-05-18T04:33:27.1337203Z 
2022-05-18T04:33:27.1337668Z ----------------------------------------------------------------------
2022-05-18T04:33:27.1338071Z Ran 1 test in 1.391s
2022-05-18T04:33:27.1338269Z 
2022-05-18T04:33:27.1338361Z OK
2022-05-18T04:33:27.1338493Z 
2022-05-18T04:33:27.1338640Z Generating XML reports...
2022-05-18T04:33:27.1370226Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043325.xml
2022-05-18T04:33:28.0673672Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:33:28.0683270Z 
2022-05-18T04:33:28.0683402Z Running tests...
2022-05-18T04:33:28.0684013Z ----------------------------------------------------------------------
2022-05-18T04:33:28.0700698Z   test_gather_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA gather (0.002s)
2022-05-18T04:33:28.0701014Z 
2022-05-18T04:33:28.0701326Z ----------------------------------------------------------------------
2022-05-18T04:33:28.0701614Z Ran 1 test in 0.002s
2022-05-18T04:33:28.0701731Z 
2022-05-18T04:33:28.0701805Z OK (skipped=1)
2022-05-18T04:33:28.0701901Z 
2022-05-18T04:33:28.0701988Z Generating XML reports...
2022-05-18T04:33:28.0733512Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043328.xml
2022-05-18T04:33:28.9076627Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:33:28.9086399Z 
2022-05-18T04:33:28.9086484Z Running tests...
2022-05-18T04:33:28.9087071Z ----------------------------------------------------------------------
2022-05-18T04:33:29.1925271Z   test_gather_full_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29287
2022-05-18T04:33:29.1947464Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29288
2022-05-18T04:33:29.1970365Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29289
2022-05-18T04:33:29.9787087Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:33:29.9853777Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:33:29.9854203Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:33:29.9854812Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:33:29.9855361Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:33:29.9887988Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:33:29.9962410Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:33:29.9962900Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:33:30.0900108Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:33:30.1076645Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:33:30.1178774Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-05-18T04:33:30.1179351Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:33:30.1180315Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:33:30.1181073Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:33:30.1181922Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:33:30.5025095Z ok (1.594s)
2022-05-18T04:33:30.5025321Z 
2022-05-18T04:33:30.5025752Z ----------------------------------------------------------------------
2022-05-18T04:33:30.5026069Z Ran 1 test in 1.594s
2022-05-18T04:33:30.5026188Z 
2022-05-18T04:33:30.5026255Z OK
2022-05-18T04:33:30.5026347Z 
2022-05-18T04:33:30.5026441Z Generating XML reports...
2022-05-18T04:33:30.5065846Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043328.xml
2022-05-18T04:33:31.4844253Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:33:31.4855002Z 
2022-05-18T04:33:31.4855320Z Running tests...
2022-05-18T04:33:31.4855924Z ----------------------------------------------------------------------
2022-05-18T04:33:31.7828289Z   test_gather_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29352
2022-05-18T04:33:31.7850915Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29353
2022-05-18T04:33:31.7874744Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29354
2022-05-18T04:33:32.6339997Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:33:32.6340685Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:33:32.6341067Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:33:32.6341928Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:33:32.6342463Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:33:32.6343151Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:33:32.6449251Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:33:32.6449760Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:33:32.6655064Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:33:32.6655652Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-05-18T04:33:32.7351074Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:33:32.7353024Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:33:32.7353764Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:33:32.7362443Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:33:32.7363343Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:33:33.0928473Z ok (1.607s)
2022-05-18T04:33:33.0928713Z 
2022-05-18T04:33:33.0929252Z ----------------------------------------------------------------------
2022-05-18T04:33:33.0929662Z Ran 1 test in 1.607s
2022-05-18T04:33:33.0929781Z 
2022-05-18T04:33:33.0929829Z OK
2022-05-18T04:33:33.0929944Z 
2022-05-18T04:33:33.0930037Z Generating XML reports...
2022-05-18T04:33:33.0962101Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043331.xml
2022-05-18T04:33:34.0183586Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:33:34.0193084Z 
2022-05-18T04:33:34.0193224Z Running tests...
2022-05-18T04:33:34.0193958Z ----------------------------------------------------------------------
2022-05-18T04:33:34.3001484Z   test_gather_object (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29413
2022-05-18T04:33:34.3024698Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29414
2022-05-18T04:33:34.3047681Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29415
2022-05-18T04:33:35.1011037Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:33:35.1112378Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:33:35.1112811Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:33:35.1113443Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:33:35.1113961Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:33:35.1114567Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:33:35.1123305Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:33:35.1124439Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:33:35.1125369Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:33:35.3097520Z ok (1.290s)
2022-05-18T04:33:35.3097722Z 
2022-05-18T04:33:35.3098225Z ----------------------------------------------------------------------
2022-05-18T04:33:35.3098532Z Ran 1 test in 1.290s
2022-05-18T04:33:35.3098648Z 
2022-05-18T04:33:35.3098711Z OK
2022-05-18T04:33:35.3098804Z 
2022-05-18T04:33:35.3098885Z Generating XML reports...
2022-05-18T04:33:35.3129060Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043334.xml
2022-05-18T04:33:36.2358847Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:33:36.2369428Z 
2022-05-18T04:33:36.2369694Z Running tests...
2022-05-18T04:33:36.2370292Z ----------------------------------------------------------------------
2022-05-18T04:33:36.5179221Z   test_gather_object_subgroup (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29466
2022-05-18T04:33:36.5201760Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29467
2022-05-18T04:33:36.5224908Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29468
2022-05-18T04:33:37.3314648Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:33:37.3415221Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:33:37.3415675Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:33:37.3416302Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:33:37.3416833Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:33:37.3417357Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:33:37.3523164Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:33:37.4429831Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:33:37.4430220Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:33:37.4840793Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:33:37.4942375Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-05-18T04:33:37.4942999Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:33:37.4943660Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:33:37.4944198Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:33:37.4944796Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:33:37.5289688Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-05-18T04:33:37.5290436Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-05-18T04:33:37.5290848Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 2
2022-05-18T04:33:37.5291463Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 3 nodes.
2022-05-18T04:33:37.5292000Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 3 nodes.
2022-05-18T04:33:37.5292546Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:3 with 3 nodes.
2022-05-18T04:33:37.5611051Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 2
2022-05-18T04:33:37.5612549Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 0
2022-05-18T04:33:37.5613122Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 1
2022-05-18T04:33:37.5613900Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:4 with 3 nodes.
2022-05-18T04:33:37.5614563Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:4 with 3 nodes.
2022-05-18T04:33:37.5615073Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:4 with 3 nodes.
2022-05-18T04:33:37.8278911Z ok (1.591s)
2022-05-18T04:33:37.8279172Z 
2022-05-18T04:33:37.8279648Z ----------------------------------------------------------------------
2022-05-18T04:33:37.8280030Z Ran 1 test in 1.591s
2022-05-18T04:33:37.8280207Z 
2022-05-18T04:33:37.8280312Z OK
2022-05-18T04:33:37.8280450Z 
2022-05-18T04:33:37.8280593Z Generating XML reports...
2022-05-18T04:33:37.8312061Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043336.xml
2022-05-18T04:33:38.7495656Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:33:38.7505510Z 
2022-05-18T04:33:38.7505610Z Running tests...
2022-05-18T04:33:38.7506421Z ----------------------------------------------------------------------
2022-05-18T04:33:39.0336690Z   test_get_backend (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29555
2022-05-18T04:33:39.0359021Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29556
2022-05-18T04:33:39.0381693Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29557
2022-05-18T04:33:39.8441559Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:33:39.8442188Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:33:39.8442652Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:33:39.8443338Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:33:39.8443855Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:33:39.8444436Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:33:39.8452462Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:33:39.8453428Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:33:39.8453800Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:33:39.8454157Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:33:39.8659257Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:33:39.8659878Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-05-18T04:33:39.8660836Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:33:39.8661727Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:33:39.8759168Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:33:40.1433493Z ok (1.392s)
2022-05-18T04:33:40.1433740Z 
2022-05-18T04:33:40.1434334Z ----------------------------------------------------------------------
2022-05-18T04:33:40.1434593Z Ran 1 test in 1.393s
2022-05-18T04:33:40.1434711Z 
2022-05-18T04:33:40.1434761Z OK
2022-05-18T04:33:40.1434852Z 
2022-05-18T04:33:40.1434945Z Generating XML reports...
2022-05-18T04:33:40.1466960Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043338.xml
2022-05-18T04:33:41.0745017Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:33:41.0756626Z 
2022-05-18T04:33:41.0756735Z Running tests...
2022-05-18T04:33:41.0757292Z ----------------------------------------------------------------------
2022-05-18T04:33:41.3610817Z   test_get_future (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29614
2022-05-18T04:33:41.3632847Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29615
2022-05-18T04:33:41.3656357Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29616
2022-05-18T04:33:42.2203380Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:33:42.2303201Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:33:42.2303906Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:33:42.2304866Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:33:42.2305398Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:33:42.2305907Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:33:42.2413812Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:33:42.2414386Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:33:42.3317201Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:33:42.5709853Z ok (1.495s)
2022-05-18T04:33:42.5710117Z 
2022-05-18T04:33:42.5710436Z ----------------------------------------------------------------------
2022-05-18T04:33:42.5710696Z Ran 1 test in 1.495s
2022-05-18T04:33:42.5710814Z 
2022-05-18T04:33:42.5710876Z OK
2022-05-18T04:33:42.5710956Z 
2022-05-18T04:33:42.5711049Z Generating XML reports...
2022-05-18T04:33:42.5743822Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043341.xml
2022-05-18T04:33:43.4986002Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:33:43.4995984Z 
2022-05-18T04:33:43.4996130Z Running tests...
2022-05-18T04:33:43.4996884Z ----------------------------------------------------------------------
2022-05-18T04:33:43.7799019Z   test_get_rank (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29667
2022-05-18T04:33:43.7821677Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29668
2022-05-18T04:33:43.7844730Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29669
2022-05-18T04:33:44.6531076Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:33:44.6553449Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:33:44.6554048Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:33:44.6554876Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:33:44.6555473Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:33:44.6632354Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:33:44.6661869Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:33:44.6662643Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:33:44.7644387Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:33:45.1901921Z ok (1.690s)
2022-05-18T04:33:45.1902145Z 
2022-05-18T04:33:45.1902667Z ----------------------------------------------------------------------
2022-05-18T04:33:45.1903186Z Ran 1 test in 1.690s
2022-05-18T04:33:45.1903292Z 
2022-05-18T04:33:45.1903352Z OK
2022-05-18T04:33:45.1903445Z 
2022-05-18T04:33:45.1903540Z Generating XML reports...
2022-05-18T04:33:45.1933908Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043343.xml
2022-05-18T04:33:46.1083523Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:33:46.1094008Z 
2022-05-18T04:33:46.1094447Z Running tests...
2022-05-18T04:33:46.1094852Z ----------------------------------------------------------------------
2022-05-18T04:33:46.3916915Z   test_get_rank_size_full_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29720
2022-05-18T04:33:46.3938953Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29721
2022-05-18T04:33:46.3961602Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29722
2022-05-18T04:33:47.2065393Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:33:47.2166503Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:33:47.2166928Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:33:47.2167555Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:33:47.2168089Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:33:47.2168604Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:33:47.2273837Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:33:47.3181506Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:33:47.3181910Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:33:47.3488219Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:33:47.3589896Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:33:47.3590479Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-05-18T04:33:47.3591129Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:33:47.3591734Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:33:47.3592253Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:33:47.6013732Z ok (1.492s)
2022-05-18T04:33:47.6013985Z 
2022-05-18T04:33:47.6014508Z ----------------------------------------------------------------------
2022-05-18T04:33:47.6014874Z Ran 1 test in 1.492s
2022-05-18T04:33:47.6015183Z 
2022-05-18T04:33:47.6015233Z OK
2022-05-18T04:33:47.6015329Z 
2022-05-18T04:33:47.6015431Z Generating XML reports...
2022-05-18T04:33:47.6048016Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043346.xml
2022-05-18T04:33:48.6035605Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:33:48.6045957Z 
2022-05-18T04:33:48.6046375Z Running tests...
2022-05-18T04:33:48.6046815Z ----------------------------------------------------------------------
2022-05-18T04:33:48.9076263Z   test_get_rank_size_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29782
2022-05-18T04:33:48.9101310Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29783
2022-05-18T04:33:48.9125625Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29784
2022-05-18T04:33:49.7373748Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:33:49.7474841Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:33:49.7475510Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:33:49.7476143Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:33:49.7476673Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:33:49.7477184Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:33:49.7582008Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:33:49.8488109Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:33:49.8488582Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:33:49.8489774Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:33:49.8695148Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:33:49.8695859Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-05-18T04:33:49.8696768Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:33:49.8697291Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:33:49.8795148Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:33:50.2180622Z ok (1.613s)
2022-05-18T04:33:50.2180866Z 
2022-05-18T04:33:50.2181365Z ----------------------------------------------------------------------
2022-05-18T04:33:50.2181781Z Ran 1 test in 1.613s
2022-05-18T04:33:50.2181896Z 
2022-05-18T04:33:50.2181956Z OK
2022-05-18T04:33:50.2182047Z 
2022-05-18T04:33:50.2182141Z Generating XML reports...
2022-05-18T04:33:50.2213225Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043348.xml
2022-05-18T04:33:51.1556756Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:33:51.1566681Z 
2022-05-18T04:33:51.1566983Z Running tests...
2022-05-18T04:33:51.1567616Z ----------------------------------------------------------------------
2022-05-18T04:33:51.1592375Z   test_invalid_static_graph (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'nccl'} (0.002s)
2022-05-18T04:33:51.1592872Z 
2022-05-18T04:33:51.1593344Z ----------------------------------------------------------------------
2022-05-18T04:33:51.1593601Z Ran 1 test in 0.002s
2022-05-18T04:33:51.1593718Z 
2022-05-18T04:33:51.1593794Z OK (skipped=1)
2022-05-18T04:33:51.1593904Z 
2022-05-18T04:33:51.1593990Z Generating XML reports...
2022-05-18T04:33:51.1624336Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043351.xml
2022-05-18T04:33:52.0023088Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:33:52.0033016Z 
2022-05-18T04:33:52.0033147Z Running tests...
2022-05-18T04:33:52.0033775Z ----------------------------------------------------------------------
2022-05-18T04:33:52.2905332Z   test_irecv (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29851
2022-05-18T04:33:52.2927617Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29852
2022-05-18T04:33:52.2950808Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29853
2022-05-18T04:33:53.1197827Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:33:53.1299889Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:33:53.1300566Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:33:53.1301466Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:33:53.1302007Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:33:53.1302568Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:33:53.1309712Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:33:53.1310706Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:33:53.1312407Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:33:53.5004277Z ok (1.497s)
2022-05-18T04:33:53.5004515Z 
2022-05-18T04:33:53.5004944Z ----------------------------------------------------------------------
2022-05-18T04:33:53.5005214Z Ran 1 test in 1.497s
2022-05-18T04:33:53.5005320Z 
2022-05-18T04:33:53.5005383Z OK
2022-05-18T04:33:53.5005476Z 
2022-05-18T04:33:53.5005570Z Generating XML reports...
2022-05-18T04:33:53.5045080Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043351.xml
2022-05-18T04:33:54.4883059Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:33:54.4893232Z 
2022-05-18T04:33:54.4893326Z Running tests...
2022-05-18T04:33:54.4894449Z ----------------------------------------------------------------------
2022-05-18T04:33:54.7857337Z   test_isend (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29904
2022-05-18T04:33:54.7881061Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29905
2022-05-18T04:33:54.7904119Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29906
2022-05-18T04:33:55.6584660Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:33:55.6686816Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:33:55.6687326Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:33:55.6688282Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:33:55.6689248Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:33:55.6690106Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:33:55.6698423Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:33:55.6699319Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:33:55.6700554Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:33:55.9957773Z ok (1.506s)
2022-05-18T04:33:55.9958046Z 
2022-05-18T04:33:55.9958558Z ----------------------------------------------------------------------
2022-05-18T04:33:55.9958827Z Ran 1 test in 1.506s
2022-05-18T04:33:55.9958942Z 
2022-05-18T04:33:55.9959005Z OK
2022-05-18T04:33:55.9959084Z 
2022-05-18T04:33:55.9959182Z Generating XML reports...
2022-05-18T04:33:55.9990493Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043354.xml
2022-05-18T04:33:57.0342838Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:33:57.0353551Z 
2022-05-18T04:33:57.0353699Z Running tests...
2022-05-18T04:33:57.0354308Z ----------------------------------------------------------------------
2022-05-18T04:33:57.3431707Z   test_isend_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29957
2022-05-18T04:33:57.3456362Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29958
2022-05-18T04:33:57.3480960Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29959
2022-05-18T04:33:58.2079952Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:33:58.2182017Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:33:58.2183212Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:33:58.2183630Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:33:58.2184138Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:33:58.2184660Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:33:58.2192849Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:33:58.2193562Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:33:58.2194743Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:33:58.5534852Z ok (1.518s)
2022-05-18T04:33:58.5535072Z 
2022-05-18T04:33:58.5535413Z ----------------------------------------------------------------------
2022-05-18T04:33:58.5535684Z Ran 1 test in 1.518s
2022-05-18T04:33:58.5535787Z 
2022-05-18T04:33:58.5535849Z OK
2022-05-18T04:33:58.5535940Z 
2022-05-18T04:33:58.5536035Z Generating XML reports...
2022-05-18T04:33:58.5568123Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043357.xml
2022-05-18T04:33:59.5932531Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:33:59.5942625Z 
2022-05-18T04:33:59.5942744Z Running tests...
2022-05-18T04:33:59.5943415Z ----------------------------------------------------------------------
2022-05-18T04:33:59.8942732Z   test_isend_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30013
2022-05-18T04:33:59.8965795Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30014
2022-05-18T04:33:59.8988233Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30015
2022-05-18T04:34:00.7510250Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:34:00.7510868Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:34:00.7511384Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:34:00.7512010Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:34:00.7512584Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:34:00.7513177Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:34:00.7520498Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:34:00.7521189Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:34:00.7523087Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:34:01.1043261Z ok (1.510s)
2022-05-18T04:34:01.1043491Z 
2022-05-18T04:34:01.1043955Z ----------------------------------------------------------------------
2022-05-18T04:34:01.1044306Z Ran 1 test in 1.510s
2022-05-18T04:34:01.1044409Z 
2022-05-18T04:34:01.1044478Z OK
2022-05-18T04:34:01.1044574Z 
2022-05-18T04:34:01.1044670Z Generating XML reports...
2022-05-18T04:34:01.1077358Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043359.xml
2022-05-18T04:34:02.1573533Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:34:02.1583966Z 
2022-05-18T04:34:02.1584138Z Running tests...
2022-05-18T04:34:02.1584747Z ----------------------------------------------------------------------
2022-05-18T04:34:02.1600998Z   test_monitored_barrier_allreduce_hang (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'nccl'} (0.002s)
2022-05-18T04:34:02.1601570Z 
2022-05-18T04:34:02.1601973Z ----------------------------------------------------------------------
2022-05-18T04:34:02.1602401Z Ran 1 test in 0.002s
2022-05-18T04:34:02.1602519Z 
2022-05-18T04:34:02.1602598Z OK (skipped=1)
2022-05-18T04:34:02.1602695Z 
2022-05-18T04:34:02.1602783Z Generating XML reports...
2022-05-18T04:34:02.1639785Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043402.xml
2022-05-18T04:34:03.0750686Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:34:03.0761398Z 
2022-05-18T04:34:03.0761707Z Running tests...
2022-05-18T04:34:03.0762389Z ----------------------------------------------------------------------
2022-05-18T04:34:03.0778257Z   test_monitored_barrier_allreduce_hang_wait_all_ranks (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'nccl'} (0.002s)
2022-05-18T04:34:03.0778772Z 
2022-05-18T04:34:03.0779146Z ----------------------------------------------------------------------
2022-05-18T04:34:03.0779590Z Ran 1 test in 0.002s
2022-05-18T04:34:03.0779780Z 
2022-05-18T04:34:03.0779908Z OK (skipped=1)
2022-05-18T04:34:03.0780034Z 
2022-05-18T04:34:03.0780122Z Generating XML reports...
2022-05-18T04:34:03.0815792Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043403.xml
2022-05-18T04:34:03.9983735Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:34:03.9994861Z 
2022-05-18T04:34:03.9994972Z Running tests...
2022-05-18T04:34:03.9995649Z ----------------------------------------------------------------------
2022-05-18T04:34:04.3088050Z   test_monitored_barrier_failure_order (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30089
2022-05-18T04:34:04.3112962Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30090
2022-05-18T04:34:04.3137720Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30091
2022-05-18T04:34:05.1766651Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:34:05.1868848Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:34:05.1869756Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:34:05.1870181Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:34:05.1870686Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:34:05.1871263Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:34:05.1879452Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:34:05.1880740Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:34:05.1881137Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:34:07.1910801Z [E ProcessGroupGloo.cpp:136] Rank 1 successfully reached monitoredBarrier, but received errors while waiting for send/recv from rank 0. Please check rank 0 logs for faulty rank.
2022-05-18T04:34:07.2010066Z [E ProcessGroupGloo.cpp:136] [Rank 0]: Rank 2 failed to pass monitoredBarrier in 2000 ms
2022-05-18T04:34:07.5220955Z ok (3.522s)
2022-05-18T04:34:07.5221181Z 
2022-05-18T04:34:07.5221688Z ----------------------------------------------------------------------
2022-05-18T04:34:07.5222043Z Ran 1 test in 3.523s
2022-05-18T04:34:07.5222157Z 
2022-05-18T04:34:07.5222218Z OK
2022-05-18T04:34:07.5222307Z 
2022-05-18T04:34:07.5222398Z Generating XML reports...
2022-05-18T04:34:07.5263240Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043403.xml
2022-05-18T04:34:08.5660066Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:34:08.5669891Z 
2022-05-18T04:34:08.5670000Z Running tests...
2022-05-18T04:34:08.5670575Z ----------------------------------------------------------------------
2022-05-18T04:34:08.8756753Z   test_monitored_barrier_gloo (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30142
2022-05-18T04:34:08.8781366Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30143
2022-05-18T04:34:08.8806343Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30144
2022-05-18T04:34:09.7086649Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:34:09.7087101Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:34:09.7087466Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:34:09.7088095Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:34:09.7088650Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:34:09.7089349Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:34:09.7099251Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:34:09.7099687Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:34:09.7100518Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:34:11.7309028Z [E ProcessGroupGloo.cpp:136] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 2000 ms
2022-05-18T04:34:11.7309574Z [E ProcessGroupGloo.cpp:136] Rank 2 successfully reached monitoredBarrier, but received errors while waiting for send/recv from rank 0. Please check rank 0 logs for faulty rank.
2022-05-18T04:34:11.9886662Z ok (3.421s)
2022-05-18T04:34:11.9886958Z 
2022-05-18T04:34:11.9887414Z ----------------------------------------------------------------------
2022-05-18T04:34:11.9887818Z Ran 1 test in 3.422s
2022-05-18T04:34:11.9887994Z 
2022-05-18T04:34:11.9888085Z OK
2022-05-18T04:34:11.9888217Z 
2022-05-18T04:34:11.9888356Z Generating XML reports...
2022-05-18T04:34:11.9920850Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043408.xml
2022-05-18T04:34:13.0373134Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:34:13.0383488Z 
2022-05-18T04:34:13.0383585Z Running tests...
2022-05-18T04:34:13.0384417Z ----------------------------------------------------------------------
2022-05-18T04:34:13.3458553Z   test_monitored_barrier_gloo_rank_0_timeout (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30195
2022-05-18T04:34:13.3482830Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30196
2022-05-18T04:34:13.3505923Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30197
2022-05-18T04:34:14.2093027Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:34:14.2102018Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:34:14.2102411Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:34:14.2103249Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:34:14.2103783Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:34:14.2104295Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:34:14.2202605Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:34:14.3108735Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:34:14.3315922Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:34:14.3316318Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:34:14.3417460Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-05-18T04:34:14.3418078Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:34:14.3418952Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:34:14.3419725Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:34:14.3420619Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:34:14.3421522Z [E ProcessGroupGloo.cpp:136] Rank 0 timed out in monitoredBarrier after 0 ms.
2022-05-18T04:34:14.3422143Z No ranks successfully processed in monitoredBarrier.
2022-05-18T04:34:14.3447640Z [E ProcessGroupGloo.cpp:136] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 0 ms
2022-05-18T04:34:14.6563174Z ok (1.618s)
2022-05-18T04:34:14.6563395Z 
2022-05-18T04:34:14.6563926Z ----------------------------------------------------------------------
2022-05-18T04:34:14.6564239Z Ran 1 test in 1.618s
2022-05-18T04:34:14.6564353Z 
2022-05-18T04:34:14.6564415Z OK
2022-05-18T04:34:14.6564505Z 
2022-05-18T04:34:14.6564584Z Generating XML reports...
2022-05-18T04:34:14.6596014Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043413.xml
2022-05-18T04:34:15.6977953Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:34:15.6988446Z 
2022-05-18T04:34:15.6988745Z Running tests...
2022-05-18T04:34:15.6989317Z ----------------------------------------------------------------------
2022-05-18T04:34:16.0041682Z   test_monitored_barrier_gloo_subgroup (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30257
2022-05-18T04:34:16.0064506Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30258
2022-05-18T04:34:16.0088248Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30259
2022-05-18T04:34:16.8614544Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:34:16.8690054Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:34:16.8690808Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:34:16.8691436Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:34:16.8691996Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:34:16.8715773Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:34:16.8800711Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:34:16.8801099Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:34:16.8804109Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-05-18T04:34:16.9728244Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:34:16.9936110Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:34:16.9936715Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:34:16.9937590Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:34:16.9938378Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:34:17.0020379Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:34:17.0025703Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:279: UserWarning: Running monitored_barrier on global rank 2 which does not belong to the given group.
2022-05-18T04:34:17.0026162Z   f"Running {op_name} on global rank {global_rank} which does not "
2022-05-18T04:34:17.1041270Z [E ProcessGroupGloo.cpp:136] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 100 ms
2022-05-18T04:34:17.4148852Z ok (1.716s)
2022-05-18T04:34:17.4149378Z 
2022-05-18T04:34:17.4149839Z ----------------------------------------------------------------------
2022-05-18T04:34:17.4150088Z Ran 1 test in 1.716s
2022-05-18T04:34:17.4150205Z 
2022-05-18T04:34:17.4150402Z OK
2022-05-18T04:34:17.4150497Z 
2022-05-18T04:34:17.4150589Z Generating XML reports...
2022-05-18T04:34:17.4183832Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043415.xml
2022-05-18T04:34:18.4840637Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:34:18.4850494Z 
2022-05-18T04:34:18.4850622Z Running tests...
2022-05-18T04:34:18.4851053Z ----------------------------------------------------------------------
2022-05-18T04:34:18.7931583Z   test_monitored_barrier_wait_all_ranks (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30316
2022-05-18T04:34:18.7955375Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30317
2022-05-18T04:34:18.7979659Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30318
2022-05-18T04:34:19.6174755Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:34:19.6188827Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:34:19.6189285Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:34:19.6189905Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:34:19.6190424Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:34:19.6276479Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:34:19.6299928Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:34:19.6300508Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:34:19.7289979Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:34:19.8487271Z [E ProcessGroupGloo.cpp:2791] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 100 ms
2022-05-18T04:34:19.8487846Z [E ProcessGroupGloo.cpp:2791] [Rank 0]: Rank 2 failed to pass monitoredBarrier in 100 ms
2022-05-18T04:34:19.8488200Z [E ProcessGroupGloo.cpp:136] [Rank 0]: Ranks 1, 2 failed to pass monitoredBarrier in 100 ms
2022-05-18T04:34:20.1034890Z ok (1.618s)
2022-05-18T04:34:20.1035155Z 
2022-05-18T04:34:20.1035605Z ----------------------------------------------------------------------
2022-05-18T04:34:20.1035860Z Ran 1 test in 1.618s
2022-05-18T04:34:20.1035976Z 
2022-05-18T04:34:20.1036039Z OK
2022-05-18T04:34:20.1036132Z 
2022-05-18T04:34:20.1036251Z Generating XML reports...
2022-05-18T04:34:20.1067642Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043418.xml
2022-05-18T04:34:21.1680590Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:34:21.1690814Z 
2022-05-18T04:34:21.1690955Z Running tests...
2022-05-18T04:34:21.1691469Z ----------------------------------------------------------------------
2022-05-18T04:34:21.1714183Z   test_nccl_backend_bool_allgather (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl'} (0.002s)
2022-05-18T04:34:21.1714616Z 
2022-05-18T04:34:21.1714961Z ----------------------------------------------------------------------
2022-05-18T04:34:21.1715336Z Ran 1 test in 0.002s
2022-05-18T04:34:21.1715516Z 
2022-05-18T04:34:21.1715642Z OK (skipped=1)
2022-05-18T04:34:21.1715814Z 
2022-05-18T04:34:21.1715950Z Generating XML reports...
2022-05-18T04:34:21.1752641Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043421.xml
2022-05-18T04:34:22.0763376Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:34:22.0772772Z 
2022-05-18T04:34:22.0772916Z Running tests...
2022-05-18T04:34:22.0773557Z ----------------------------------------------------------------------
2022-05-18T04:34:22.0797821Z   test_nccl_backend_bool_allreduce (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl'} (0.002s)
2022-05-18T04:34:22.0798239Z 
2022-05-18T04:34:22.0798449Z ----------------------------------------------------------------------
2022-05-18T04:34:22.0798680Z Ran 1 test in 0.002s
2022-05-18T04:34:22.0798795Z 
2022-05-18T04:34:22.0798884Z OK (skipped=1)
2022-05-18T04:34:22.0799039Z 
2022-05-18T04:34:22.0799125Z Generating XML reports...
2022-05-18T04:34:22.0834889Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043422.xml
2022-05-18T04:34:22.9927899Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:34:22.9938075Z 
2022-05-18T04:34:22.9938283Z Running tests...
2022-05-18T04:34:22.9938620Z ----------------------------------------------------------------------
2022-05-18T04:34:22.9959595Z   test_nccl_backend_bool_broadcast (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl'} (0.002s)
2022-05-18T04:34:22.9959943Z 
2022-05-18T04:34:22.9960197Z ----------------------------------------------------------------------
2022-05-18T04:34:22.9960449Z Ran 1 test in 0.002s
2022-05-18T04:34:22.9960566Z 
2022-05-18T04:34:22.9960639Z OK (skipped=1)
2022-05-18T04:34:22.9960752Z 
2022-05-18T04:34:22.9960876Z Generating XML reports...
2022-05-18T04:34:23.0002154Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043422.xml
2022-05-18T04:34:23.9071301Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:34:23.9082217Z 
2022-05-18T04:34:23.9082521Z Running tests...
2022-05-18T04:34:23.9083117Z ----------------------------------------------------------------------
2022-05-18T04:34:23.9108464Z   test_nccl_backend_bool_reduce (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl'} (0.002s)
2022-05-18T04:34:23.9108781Z 
2022-05-18T04:34:23.9109135Z ----------------------------------------------------------------------
2022-05-18T04:34:23.9109586Z Ran 1 test in 0.003s
2022-05-18T04:34:23.9109802Z 
2022-05-18T04:34:23.9109925Z OK (skipped=1)
2022-05-18T04:34:23.9110041Z 
2022-05-18T04:34:23.9110128Z Generating XML reports...
2022-05-18T04:34:23.9145625Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043423.xml
2022-05-18T04:34:24.8080200Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:34:24.8090474Z 
2022-05-18T04:34:24.8090752Z Running tests...
2022-05-18T04:34:24.8091377Z ----------------------------------------------------------------------
2022-05-18T04:34:24.8112354Z   test_nccl_high_priority_stream (__main__.TestDistBackendWithSpawn) ... skip: Only NCCL backend supports high priority stream (0.002s)
2022-05-18T04:34:24.8112753Z 
2022-05-18T04:34:24.8113145Z ----------------------------------------------------------------------
2022-05-18T04:34:24.8113572Z Ran 1 test in 0.002s
2022-05-18T04:34:24.8113705Z 
2022-05-18T04:34:24.8113782Z OK (skipped=1)
2022-05-18T04:34:24.8113889Z 
2022-05-18T04:34:24.8113975Z Generating XML reports...
2022-05-18T04:34:24.8148123Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043424.xml
2022-05-18T04:34:25.7185944Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:34:25.7196073Z 
2022-05-18T04:34:25.7196186Z Running tests...
2022-05-18T04:34:25.7196767Z ----------------------------------------------------------------------
2022-05-18T04:34:25.7215467Z   test_new_subgroups (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2022-05-18T04:34:25.7215818Z 
2022-05-18T04:34:25.7216116Z ----------------------------------------------------------------------
2022-05-18T04:34:25.7216429Z Ran 1 test in 0.002s
2022-05-18T04:34:25.7216548Z 
2022-05-18T04:34:25.7216609Z OK (skipped=1)
2022-05-18T04:34:25.7216718Z 
2022-05-18T04:34:25.7216804Z Generating XML reports...
2022-05-18T04:34:25.7251920Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043425.xml
2022-05-18T04:34:26.6301595Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:34:26.6312224Z 
2022-05-18T04:34:26.6312649Z Running tests...
2022-05-18T04:34:26.6313277Z ----------------------------------------------------------------------
2022-05-18T04:34:26.6334719Z   test_new_subgroups_by_enumeration (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2022-05-18T04:34:26.6335022Z 
2022-05-18T04:34:26.6335326Z ----------------------------------------------------------------------
2022-05-18T04:34:26.6335657Z Ran 1 test in 0.002s
2022-05-18T04:34:26.6335779Z 
2022-05-18T04:34:26.6335861Z OK (skipped=1)
2022-05-18T04:34:26.6335971Z 
2022-05-18T04:34:26.6336046Z Generating XML reports...
2022-05-18T04:34:26.6371776Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043426.xml
2022-05-18T04:34:27.5399006Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:34:27.5409284Z 
2022-05-18T04:34:27.5409390Z Running tests...
2022-05-18T04:34:27.5409811Z ----------------------------------------------------------------------
2022-05-18T04:34:27.5428790Z   test_new_subgroups_by_enumeration_input_rank_exceeds_world_size (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2022-05-18T04:34:27.5429351Z 
2022-05-18T04:34:27.5429933Z ----------------------------------------------------------------------
2022-05-18T04:34:27.5430396Z Ran 1 test in 0.002s
2022-05-18T04:34:27.5430523Z 
2022-05-18T04:34:27.5430602Z OK (skipped=1)
2022-05-18T04:34:27.5430715Z 
2022-05-18T04:34:27.5430861Z Generating XML reports...
2022-05-18T04:34:27.5464359Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043427.xml
2022-05-18T04:34:28.4502458Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:34:28.4512614Z 
2022-05-18T04:34:28.4512769Z Running tests...
2022-05-18T04:34:28.4513504Z ----------------------------------------------------------------------
2022-05-18T04:34:28.7562333Z   test_new_subgroups_by_enumeration_negative_input_rank (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30449
2022-05-18T04:34:28.7586141Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30450
2022-05-18T04:34:28.7611139Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30451
2022-05-18T04:34:29.6020954Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:34:29.6123088Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:34:29.6123787Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:34:29.6124394Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:34:29.6124932Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:34:29.6125639Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:34:29.6135269Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:34:29.6135874Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:34:29.6136405Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:34:29.8663603Z skip: CUDA is not available. (1.415s)
2022-05-18T04:34:29.8663928Z 
2022-05-18T04:34:29.8664379Z ----------------------------------------------------------------------
2022-05-18T04:34:29.8664638Z Ran 1 test in 1.415s
2022-05-18T04:34:29.8664753Z 
2022-05-18T04:34:29.8664826Z OK (skipped=1)
2022-05-18T04:34:29.8664935Z 
2022-05-18T04:34:29.8665022Z Generating XML reports...
2022-05-18T04:34:29.8696802Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043428.xml
2022-05-18T04:34:30.9045126Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:34:30.9055339Z 
2022-05-18T04:34:30.9055450Z Running tests...
2022-05-18T04:34:30.9055929Z ----------------------------------------------------------------------
2022-05-18T04:34:31.2088716Z   test_new_subgroups_group_size_exceeds_world_size (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30502
2022-05-18T04:34:31.2111491Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30503
2022-05-18T04:34:31.2136253Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30504
2022-05-18T04:34:32.0698265Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:34:32.0698699Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:34:32.0699087Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:34:32.0699698Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:34:32.0700225Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:34:32.0700746Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:34:32.0710498Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:34:32.0711106Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:34:32.0713894Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:34:32.3190427Z skip: CUDA is not available. (1.413s)
2022-05-18T04:34:32.3190796Z 
2022-05-18T04:34:32.3191314Z ----------------------------------------------------------------------
2022-05-18T04:34:32.3191680Z Ran 1 test in 1.413s
2022-05-18T04:34:32.3191796Z 
2022-05-18T04:34:32.3191877Z OK (skipped=1)
2022-05-18T04:34:32.3191974Z 
2022-05-18T04:34:32.3192060Z Generating XML reports...
2022-05-18T04:34:32.3224381Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043430.xml
2022-05-18T04:34:33.3433757Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:34:33.3444409Z 
2022-05-18T04:34:33.3444874Z Running tests...
2022-05-18T04:34:33.3445493Z ----------------------------------------------------------------------
2022-05-18T04:34:33.3462093Z   test_new_subgroups_overlap_not_allowed (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2022-05-18T04:34:33.3462456Z 
2022-05-18T04:34:33.3462812Z ----------------------------------------------------------------------
2022-05-18T04:34:33.3463470Z Ran 1 test in 0.002s
2022-05-18T04:34:33.3463586Z 
2022-05-18T04:34:33.3463661Z OK (skipped=1)
2022-05-18T04:34:33.3463774Z 
2022-05-18T04:34:33.3463930Z Generating XML reports...
2022-05-18T04:34:33.3499709Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043433.xml
2022-05-18T04:34:34.2485698Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:34:34.2495545Z 
2022-05-18T04:34:34.2495999Z Running tests...
2022-05-18T04:34:34.2496467Z ----------------------------------------------------------------------
2022-05-18T04:34:34.2512863Z   test_new_subgroups_world_size_not_divisible_by_group_size (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2022-05-18T04:34:34.2513284Z 
2022-05-18T04:34:34.2513558Z ----------------------------------------------------------------------
2022-05-18T04:34:34.2513828Z Ran 1 test in 0.002s
2022-05-18T04:34:34.2513984Z 
2022-05-18T04:34:34.2514070Z OK (skipped=1)
2022-05-18T04:34:34.2514181Z 
2022-05-18T04:34:34.2514253Z Generating XML reports...
2022-05-18T04:34:34.2549871Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043434.xml
2022-05-18T04:34:35.1558151Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:34:35.1568874Z 
2022-05-18T04:34:35.1568979Z Running tests...
2022-05-18T04:34:35.1569597Z ----------------------------------------------------------------------
2022-05-18T04:34:35.4603239Z   test_output_unused_in_loss_dict_module (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30575
2022-05-18T04:34:35.4627267Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30576
2022-05-18T04:34:35.4651681Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30577
2022-05-18T04:34:36.3196453Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:34:36.3265696Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:34:36.3266401Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:34:36.3267045Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:34:36.3267573Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:34:36.3297895Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:34:36.3377535Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:34:36.3377941Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:34:36.4310856Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:34:36.6707419Z skip: Need at least 2 CUDA devices (1.514s)
2022-05-18T04:34:36.6707712Z 
2022-05-18T04:34:36.6708078Z ----------------------------------------------------------------------
2022-05-18T04:34:36.6708348Z Ran 1 test in 1.514s
2022-05-18T04:34:36.6708462Z 
2022-05-18T04:34:36.6708536Z OK (skipped=1)
2022-05-18T04:34:36.6708644Z 
2022-05-18T04:34:36.6708716Z Generating XML reports...
2022-05-18T04:34:36.6739969Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043435.xml
2022-05-18T04:34:37.6940028Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:34:37.6951067Z 
2022-05-18T04:34:37.6951206Z Running tests...
2022-05-18T04:34:37.6951663Z ----------------------------------------------------------------------
2022-05-18T04:34:37.9940232Z   test_output_unused_in_loss_tuple_module (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30628
2022-05-18T04:34:37.9964295Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30629
2022-05-18T04:34:37.9989169Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30630
2022-05-18T04:34:38.8571557Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:34:38.8673849Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:34:38.8674489Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:34:38.8675403Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:34:38.8676299Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:34:38.8676842Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:34:38.8685203Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:34:38.8685763Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:34:38.8687366Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:34:39.1041139Z skip: Need at least 2 CUDA devices (1.409s)
2022-05-18T04:34:39.1041433Z 
2022-05-18T04:34:39.1041903Z ----------------------------------------------------------------------
2022-05-18T04:34:39.1042218Z Ran 1 test in 1.409s
2022-05-18T04:34:39.1042349Z 
2022-05-18T04:34:39.1042424Z OK (skipped=1)
2022-05-18T04:34:39.1042533Z 
2022-05-18T04:34:39.1042637Z Generating XML reports...
2022-05-18T04:34:39.1073661Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043437.xml
2022-05-18T04:34:40.1075839Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:34:40.1085555Z 
2022-05-18T04:34:40.1085694Z Running tests...
2022-05-18T04:34:40.1086303Z ----------------------------------------------------------------------
2022-05-18T04:34:40.4048542Z   test_periodic_model_averager (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30681
2022-05-18T04:34:40.4072362Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30682
2022-05-18T04:34:40.4095225Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30683
2022-05-18T04:34:41.2616428Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:34:41.2617120Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:34:41.2617705Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:34:41.2618420Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:34:41.2618950Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:34:41.2619492Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:34:41.2627326Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:34:41.2627733Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:34:41.2629854Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:34:41.5146432Z skip: Need at least 2 CUDA devices (1.406s)
2022-05-18T04:34:41.5146751Z 
2022-05-18T04:34:41.5147508Z ----------------------------------------------------------------------
2022-05-18T04:34:41.5147996Z Ran 1 test in 1.406s
2022-05-18T04:34:41.5148207Z 
2022-05-18T04:34:41.5148343Z OK (skipped=1)
2022-05-18T04:34:41.5148529Z 
2022-05-18T04:34:41.5148623Z Generating XML reports...
2022-05-18T04:34:41.5178466Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043440.xml
2022-05-18T04:34:42.4400489Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:34:42.4409947Z 
2022-05-18T04:34:42.4410089Z Running tests...
2022-05-18T04:34:42.4410543Z ----------------------------------------------------------------------
2022-05-18T04:34:42.7213668Z   test_periodic_model_averager_param_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30734
2022-05-18T04:34:42.7235937Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30735
2022-05-18T04:34:42.7258843Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30736
2022-05-18T04:34:43.5573721Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:34:43.5674842Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:34:43.5675582Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:34:43.5676180Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:34:43.5676714Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:34:43.5677235Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:34:43.5782482Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:34:43.6690156Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:34:43.6690964Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:34:43.8307792Z skip: Need at least 2 CUDA devices (1.389s)
2022-05-18T04:34:43.8308033Z 
2022-05-18T04:34:43.8308456Z ----------------------------------------------------------------------
2022-05-18T04:34:43.8308714Z Ran 1 test in 1.390s
2022-05-18T04:34:43.8308816Z 
2022-05-18T04:34:43.8308893Z OK (skipped=1)
2022-05-18T04:34:43.8309002Z 
2022-05-18T04:34:43.8310230Z Generating XML reports...
2022-05-18T04:34:43.8342456Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043442.xml
2022-05-18T04:34:44.7471471Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:34:44.7480579Z 
2022-05-18T04:34:44.7480692Z Running tests...
2022-05-18T04:34:44.7481152Z ----------------------------------------------------------------------
2022-05-18T04:34:45.0218496Z   test_post_localSGD_optimizer_parity (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77123 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure IN_CI is not set and you are not using the flag --import-disabled-tests. (0.273s)
2022-05-18T04:34:45.0219148Z 
2022-05-18T04:34:45.0219358Z ----------------------------------------------------------------------
2022-05-18T04:34:45.0219624Z Ran 1 test in 0.274s
2022-05-18T04:34:45.0219782Z 
2022-05-18T04:34:45.0219858Z OK (skipped=1)
2022-05-18T04:34:45.0219965Z 
2022-05-18T04:34:45.0220266Z Generating XML reports...
2022-05-18T04:34:45.0246744Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043444.xml
2022-05-18T04:34:45.9128249Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:34:45.9137443Z 
2022-05-18T04:34:45.9137576Z Running tests...
2022-05-18T04:34:45.9138302Z ----------------------------------------------------------------------
2022-05-18T04:34:46.1848071Z   test_post_localSGD_optimizer_parity_grad_is_view (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77292 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure IN_CI is not set and you are not using the flag --import-disabled-tests. (0.271s)
2022-05-18T04:34:46.1849044Z 
2022-05-18T04:34:46.1849432Z ----------------------------------------------------------------------
2022-05-18T04:34:46.1849865Z Ran 1 test in 0.271s
2022-05-18T04:34:46.1850050Z 
2022-05-18T04:34:46.1850156Z OK (skipped=1)
2022-05-18T04:34:46.1850336Z 
2022-05-18T04:34:46.1850489Z Generating XML reports...
2022-05-18T04:34:46.1878375Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043445.xml
2022-05-18T04:34:47.0789166Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:34:47.0799027Z 
2022-05-18T04:34:47.0799458Z Running tests...
2022-05-18T04:34:47.0799845Z ----------------------------------------------------------------------
2022-05-18T04:34:47.3617256Z   test_post_localSGD_optimizer_parity_with_hierarchical_sgd (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30807
2022-05-18T04:34:47.3639868Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30808
2022-05-18T04:34:47.3662401Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30809
2022-05-18T04:34:48.1493507Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:34:48.1586944Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:34:48.1587552Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:34:48.1588186Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:34:48.1588704Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:34:48.1594576Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:34:48.1696179Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:34:48.1697095Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:34:48.2607156Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:34:48.3711202Z skip: Need at least 4 CUDA devices (1.291s)
2022-05-18T04:34:48.3711510Z 
2022-05-18T04:34:48.3711913Z ----------------------------------------------------------------------
2022-05-18T04:34:48.3712186Z Ran 1 test in 1.291s
2022-05-18T04:34:48.3712301Z 
2022-05-18T04:34:48.3712380Z OK (skipped=1)
2022-05-18T04:34:48.3712491Z 
2022-05-18T04:34:48.3712564Z Generating XML reports...
2022-05-18T04:34:48.3742765Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043447.xml
2022-05-18T04:34:49.3148334Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:34:49.3159186Z 
2022-05-18T04:34:49.3159795Z Running tests...
2022-05-18T04:34:49.3160376Z ----------------------------------------------------------------------
2022-05-18T04:34:49.6222372Z   test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30860
2022-05-18T04:34:49.6246495Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30861
2022-05-18T04:34:49.6271310Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30862
2022-05-18T04:34:50.4943196Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:34:50.5044675Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:34:50.5045536Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:34:50.5045949Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:34:50.5046480Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:34:50.5047002Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:34:50.5056225Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:34:50.5057399Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:34:50.5057823Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:34:50.7324868Z skip: Need at least 4 CUDA devices (1.416s)
2022-05-18T04:34:50.7325114Z 
2022-05-18T04:34:50.7325414Z ----------------------------------------------------------------------
2022-05-18T04:34:50.7325664Z Ran 1 test in 1.416s
2022-05-18T04:34:50.7325782Z 
2022-05-18T04:34:50.7325859Z OK (skipped=1)
2022-05-18T04:34:50.7325967Z 
2022-05-18T04:34:50.7326053Z Generating XML reports...
2022-05-18T04:34:50.7357041Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043449.xml
2022-05-18T04:34:51.7355054Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:34:51.7365808Z 
2022-05-18T04:34:51.7366210Z Running tests...
2022-05-18T04:34:51.7366638Z ----------------------------------------------------------------------
2022-05-18T04:34:52.0383434Z   test_reduce_full_group_max (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30913
2022-05-18T04:34:52.0407346Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30914
2022-05-18T04:34:52.0431518Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30915
2022-05-18T04:34:52.8812923Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:34:52.8914656Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:34:52.8915074Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:34:52.8915700Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:34:52.8916232Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:34:52.8916764Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:34:52.8925698Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:34:52.8927032Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:34:52.8927862Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:34:52.9035760Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:34:52.9137629Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-05-18T04:34:52.9138258Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:34:52.9139208Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:34:52.9139971Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:34:52.9140489Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:34:53.2486011Z ok (1.512s)
2022-05-18T04:34:53.2486266Z 
2022-05-18T04:34:53.2486806Z ----------------------------------------------------------------------
2022-05-18T04:34:53.2487188Z Ran 1 test in 1.512s
2022-05-18T04:34:53.2487291Z 
2022-05-18T04:34:53.2487360Z OK
2022-05-18T04:34:53.2487452Z 
2022-05-18T04:34:53.2487546Z Generating XML reports...
2022-05-18T04:34:53.2519204Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043451.xml
2022-05-18T04:34:54.2798825Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:34:54.2808927Z 
2022-05-18T04:34:54.2809202Z Running tests...
2022-05-18T04:34:54.2809777Z ----------------------------------------------------------------------
2022-05-18T04:34:54.5827991Z   test_reduce_full_group_min (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30978
2022-05-18T04:34:54.5851080Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30979
2022-05-18T04:34:54.5875088Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30980
2022-05-18T04:34:55.4461094Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:34:55.4562695Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:34:55.4563165Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:34:55.4563890Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:34:55.4564429Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:34:55.4564937Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:34:55.4672567Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:34:55.5578116Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:34:55.5578533Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:34:55.5787502Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:34:55.5788228Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:34:55.5788635Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-05-18T04:34:55.5789280Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:34:55.5789807Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:34:55.5790590Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:34:55.8930979Z ok (1.612s)
2022-05-18T04:34:55.8931459Z 
2022-05-18T04:34:55.8932015Z ----------------------------------------------------------------------
2022-05-18T04:34:55.8932350Z Ran 1 test in 1.612s
2022-05-18T04:34:55.8932467Z 
2022-05-18T04:34:55.8932529Z OK
2022-05-18T04:34:55.8932622Z 
2022-05-18T04:34:55.8932705Z Generating XML reports...
2022-05-18T04:34:55.8963672Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043454.xml
2022-05-18T04:34:56.9218767Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:34:56.9229031Z 
2022-05-18T04:34:56.9229181Z Running tests...
2022-05-18T04:34:56.9229802Z ----------------------------------------------------------------------
2022-05-18T04:34:57.2233993Z   test_reduce_full_group_product (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31043
2022-05-18T04:34:57.2258507Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31044
2022-05-18T04:34:57.2281889Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31045
2022-05-18T04:34:58.0464591Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:34:58.0562617Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:34:58.0563178Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:34:58.0564059Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:34:58.0564866Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:34:58.0565670Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:34:58.0675210Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:34:58.0675808Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:34:58.1580031Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:34:58.1789549Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-05-18T04:34:58.1790208Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:34:58.1790849Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:34:58.1791611Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:34:58.1792168Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:34:58.1792778Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:34:58.4336588Z ok (1.510s)
2022-05-18T04:34:58.4336876Z 
2022-05-18T04:34:58.4337353Z ----------------------------------------------------------------------
2022-05-18T04:34:58.4337798Z Ran 1 test in 1.511s
2022-05-18T04:34:58.4337976Z 
2022-05-18T04:34:58.4338038Z OK
2022-05-18T04:34:58.4338136Z 
2022-05-18T04:34:58.4338228Z Generating XML reports...
2022-05-18T04:34:58.4369487Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043456.xml
2022-05-18T04:34:59.4434193Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:34:59.4443546Z 
2022-05-18T04:34:59.4443689Z Running tests...
2022-05-18T04:34:59.4444571Z ----------------------------------------------------------------------
2022-05-18T04:34:59.7416716Z   test_reduce_full_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31108
2022-05-18T04:34:59.7438967Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31109
2022-05-18T04:34:59.7462361Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31110
2022-05-18T04:35:00.5872897Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:35:00.5973822Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:35:00.5974513Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:35:00.5975343Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:35:00.5975886Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:35:00.5976399Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:35:00.5984770Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:35:00.5986512Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:35:00.5988260Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:35:00.6192734Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:35:00.6294472Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-05-18T04:35:00.6294977Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:35:00.6295679Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:35:00.6296194Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:35:00.6296720Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:35:00.9515490Z ok (1.507s)
2022-05-18T04:35:00.9515739Z 
2022-05-18T04:35:00.9516204Z ----------------------------------------------------------------------
2022-05-18T04:35:00.9516585Z Ran 1 test in 1.507s
2022-05-18T04:35:00.9516771Z 
2022-05-18T04:35:00.9516877Z OK
2022-05-18T04:35:00.9517007Z 
2022-05-18T04:35:00.9517153Z Generating XML reports...
2022-05-18T04:35:00.9548586Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043459.xml
2022-05-18T04:35:01.8986545Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:35:01.8997018Z 
2022-05-18T04:35:01.8997139Z Running tests...
2022-05-18T04:35:01.8998015Z ----------------------------------------------------------------------
2022-05-18T04:35:02.1822181Z   test_reduce_group_max (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31173
2022-05-18T04:35:02.1844703Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31174
2022-05-18T04:35:02.1867326Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31175
2022-05-18T04:35:03.0075812Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:35:03.0076441Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:35:03.0077172Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:35:03.0078173Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:35:03.0078985Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:35:03.0079782Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:35:03.0164317Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:35:03.0164837Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:35:03.0370588Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:35:03.0371009Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-05-18T04:35:03.1067595Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:35:03.1069643Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:35:03.1070466Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:35:03.1078367Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:35:03.1078898Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:35:03.3921272Z ok (1.492s)
2022-05-18T04:35:03.3921498Z 
2022-05-18T04:35:03.3921895Z ----------------------------------------------------------------------
2022-05-18T04:35:03.3922163Z Ran 1 test in 1.492s
2022-05-18T04:35:03.3922280Z 
2022-05-18T04:35:03.3922359Z OK
2022-05-18T04:35:03.3922438Z 
2022-05-18T04:35:03.3922529Z Generating XML reports...
2022-05-18T04:35:03.3954213Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043501.xml
2022-05-18T04:35:04.3192632Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:35:04.3202474Z 
2022-05-18T04:35:04.3202667Z Running tests...
2022-05-18T04:35:04.3203086Z ----------------------------------------------------------------------
2022-05-18T04:35:04.6020314Z   test_reduce_group_min (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31234
2022-05-18T04:35:04.6041913Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31235
2022-05-18T04:35:04.6065416Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31236
2022-05-18T04:35:05.4021855Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:35:05.4022390Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:35:05.4022825Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:35:05.4023682Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:35:05.4024267Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:35:05.4024782Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:35:05.4030968Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:35:05.4032972Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:35:05.4033705Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:35:05.4034065Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:35:05.4239977Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:35:05.4240618Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-05-18T04:35:05.4241276Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:35:05.4241795Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:35:05.4336623Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:35:05.7118718Z ok (1.391s)
2022-05-18T04:35:05.7118988Z 
2022-05-18T04:35:05.7119555Z ----------------------------------------------------------------------
2022-05-18T04:35:05.7119793Z Ran 1 test in 1.391s
2022-05-18T04:35:05.7119910Z 
2022-05-18T04:35:05.7119976Z OK
2022-05-18T04:35:05.7120068Z 
2022-05-18T04:35:05.7120171Z Generating XML reports...
2022-05-18T04:35:05.7150184Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043504.xml
2022-05-18T04:35:06.6862657Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:35:06.6873110Z 
2022-05-18T04:35:06.6873244Z Running tests...
2022-05-18T04:35:06.6873842Z ----------------------------------------------------------------------
2022-05-18T04:35:06.9800590Z   test_reduce_group_product (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31295
2022-05-18T04:35:06.9824335Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31296
2022-05-18T04:35:06.9847938Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31297
2022-05-18T04:35:07.7969205Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:35:07.7969694Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:35:07.7970056Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:35:07.7970682Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:35:07.7971211Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:35:07.7971737Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:35:07.7981647Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:35:07.7982352Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:35:07.7983148Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:35:07.7985421Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:35:07.8192445Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:35:07.8193043Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-05-18T04:35:07.8193678Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:35:07.8194214Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:35:07.8291797Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:35:08.1900770Z ok (1.502s)
2022-05-18T04:35:08.1900974Z 
2022-05-18T04:35:08.1901526Z ----------------------------------------------------------------------
2022-05-18T04:35:08.1901795Z Ran 1 test in 1.503s
2022-05-18T04:35:08.1901900Z 
2022-05-18T04:35:08.1901962Z OK
2022-05-18T04:35:08.1902053Z 
2022-05-18T04:35:08.1902149Z Generating XML reports...
2022-05-18T04:35:08.1933212Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043506.xml
2022-05-18T04:35:09.1593764Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:35:09.1603685Z 
2022-05-18T04:35:09.1603813Z Running tests...
2022-05-18T04:35:09.1604619Z ----------------------------------------------------------------------
2022-05-18T04:35:09.4474360Z   test_reduce_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31356
2022-05-18T04:35:09.4499760Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31357
2022-05-18T04:35:09.4524349Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31358
2022-05-18T04:35:10.2978081Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:35:10.3079799Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:35:10.3080278Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:35:10.3080916Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:35:10.3081450Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:35:10.3081957Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:35:10.3092907Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:35:10.3093541Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:35:10.3093898Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:35:10.3096821Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:35:10.3303572Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-05-18T04:35:10.3304115Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:35:10.3304738Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:35:10.3305282Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:35:10.3402145Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:35:10.6577385Z ok (1.497s)
2022-05-18T04:35:10.6577921Z 
2022-05-18T04:35:10.6578883Z ----------------------------------------------------------------------
2022-05-18T04:35:10.6579321Z Ran 1 test in 1.497s
2022-05-18T04:35:10.6579441Z 
2022-05-18T04:35:10.6579512Z OK
2022-05-18T04:35:10.6579605Z 
2022-05-18T04:35:10.6579701Z Generating XML reports...
2022-05-18T04:35:10.6609669Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043509.xml
2022-05-18T04:35:11.6385587Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:35:11.6395038Z 
2022-05-18T04:35:11.6395182Z Running tests...
2022-05-18T04:35:11.6396179Z ----------------------------------------------------------------------
2022-05-18T04:35:11.9335992Z   test_reduce_max (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31417
2022-05-18T04:35:11.9359826Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31418
2022-05-18T04:35:11.9383774Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31419
2022-05-18T04:35:12.7451696Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:35:12.7552892Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:35:12.7553543Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:35:12.7554172Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:35:12.7554702Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:35:12.7555231Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:35:12.7661646Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:35:12.8566812Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:35:12.8567322Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:35:13.1439198Z ok (1.504s)
2022-05-18T04:35:13.1439432Z 
2022-05-18T04:35:13.1439878Z ----------------------------------------------------------------------
2022-05-18T04:35:13.1440283Z Ran 1 test in 1.504s
2022-05-18T04:35:13.1440465Z 
2022-05-18T04:35:13.1440564Z OK
2022-05-18T04:35:13.1440702Z 
2022-05-18T04:35:13.1440840Z Generating XML reports...
2022-05-18T04:35:13.1472161Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043511.xml
2022-05-18T04:35:14.0900698Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:35:14.0910728Z 
2022-05-18T04:35:14.0911037Z Running tests...
2022-05-18T04:35:14.0911649Z ----------------------------------------------------------------------
2022-05-18T04:35:14.3773765Z   test_reduce_min (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31473
2022-05-18T04:35:14.3796355Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31474
2022-05-18T04:35:14.3818630Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31475
2022-05-18T04:35:15.2204586Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:35:15.2205262Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:35:15.2205666Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:35:15.2206299Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:35:15.2206833Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:35:15.2207342Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:35:15.2214998Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:35:15.2216147Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:35:15.2216590Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:35:15.4870654Z ok (1.396s)
2022-05-18T04:35:15.4870952Z 
2022-05-18T04:35:15.4871474Z ----------------------------------------------------------------------
2022-05-18T04:35:15.4871731Z Ran 1 test in 1.396s
2022-05-18T04:35:15.4872046Z 
2022-05-18T04:35:15.4872110Z OK
2022-05-18T04:35:15.4872206Z 
2022-05-18T04:35:15.4872300Z Generating XML reports...
2022-05-18T04:35:15.4904276Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043514.xml
2022-05-18T04:35:16.4440191Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:35:16.4449042Z 
2022-05-18T04:35:16.4449178Z Running tests...
2022-05-18T04:35:16.4449621Z ----------------------------------------------------------------------
2022-05-18T04:35:16.4469196Z   test_reduce_multigpu (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl backend supports reduce multigpu (0.002s)
2022-05-18T04:35:16.4469499Z 
2022-05-18T04:35:16.4469864Z ----------------------------------------------------------------------
2022-05-18T04:35:16.4470106Z Ran 1 test in 0.002s
2022-05-18T04:35:16.4470275Z 
2022-05-18T04:35:16.4470351Z OK (skipped=1)
2022-05-18T04:35:16.4470466Z 
2022-05-18T04:35:16.4470558Z Generating XML reports...
2022-05-18T04:35:16.4501586Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043516.xml
2022-05-18T04:35:17.2961885Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:35:17.2971734Z 
2022-05-18T04:35:17.2972145Z Running tests...
2022-05-18T04:35:17.2972719Z ----------------------------------------------------------------------
2022-05-18T04:35:17.5840125Z   test_reduce_product (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31539
2022-05-18T04:35:17.5862698Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31540
2022-05-18T04:35:17.5886011Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31541
2022-05-18T04:35:18.3926316Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:35:18.4026818Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:35:18.4027419Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:35:18.4028090Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:35:18.4028619Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:35:18.4029292Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:35:18.4037562Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:35:18.4038799Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:35:18.4039195Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:35:18.6938130Z ok (1.396s)
2022-05-18T04:35:18.6938311Z 
2022-05-18T04:35:18.6938724Z ----------------------------------------------------------------------
2022-05-18T04:35:18.6939032Z Ran 1 test in 1.396s
2022-05-18T04:35:18.6939150Z 
2022-05-18T04:35:18.6939218Z OK
2022-05-18T04:35:18.6939297Z 
2022-05-18T04:35:18.6939393Z Generating XML reports...
2022-05-18T04:35:18.6970227Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043517.xml
2022-05-18T04:35:19.6381639Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:35:19.6393005Z 
2022-05-18T04:35:19.6393512Z Running tests...
2022-05-18T04:35:19.6394143Z ----------------------------------------------------------------------
2022-05-18T04:35:19.9281793Z   test_reduce_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31595
2022-05-18T04:35:19.9303745Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31596
2022-05-18T04:35:19.9326777Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31597
2022-05-18T04:35:20.7770811Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:35:20.7771577Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:35:20.7772124Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:35:20.7772749Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:35:20.7773294Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:35:20.7773824Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:35:20.7877895Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:35:20.8784553Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:35:20.8784978Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:35:21.1380637Z ok (1.498s)
2022-05-18T04:35:21.1380903Z 
2022-05-18T04:35:21.1381365Z ----------------------------------------------------------------------
2022-05-18T04:35:21.1381769Z Ran 1 test in 1.499s
2022-05-18T04:35:21.1381943Z 
2022-05-18T04:35:21.1382038Z OK
2022-05-18T04:35:21.1382162Z 
2022-05-18T04:35:21.1382311Z Generating XML reports...
2022-05-18T04:35:21.1414062Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043519.xml
2022-05-18T04:35:22.0753201Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:35:22.0763422Z 
2022-05-18T04:35:22.0763515Z Running tests...
2022-05-18T04:35:22.0764245Z ----------------------------------------------------------------------
2022-05-18T04:35:22.0782524Z   test_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA reduce (0.002s)
2022-05-18T04:35:22.0783141Z 
2022-05-18T04:35:22.0783533Z ----------------------------------------------------------------------
2022-05-18T04:35:22.0783942Z Ran 1 test in 0.002s
2022-05-18T04:35:22.0784121Z 
2022-05-18T04:35:22.0784240Z OK (skipped=1)
2022-05-18T04:35:22.0784414Z 
2022-05-18T04:35:22.0784548Z Generating XML reports...
2022-05-18T04:35:22.0817377Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043522.xml
2022-05-18T04:35:22.9236579Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:35:22.9245736Z 
2022-05-18T04:35:22.9245885Z Running tests...
2022-05-18T04:35:22.9246476Z ----------------------------------------------------------------------
2022-05-18T04:35:22.9264243Z   test_reduce_sum_cuda_twice (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA reduce (0.002s)
2022-05-18T04:35:22.9264560Z 
2022-05-18T04:35:22.9264929Z ----------------------------------------------------------------------
2022-05-18T04:35:22.9265379Z Ran 1 test in 0.002s
2022-05-18T04:35:22.9265564Z 
2022-05-18T04:35:22.9265642Z OK (skipped=1)
2022-05-18T04:35:22.9265738Z 
2022-05-18T04:35:22.9265826Z Generating XML reports...
2022-05-18T04:35:22.9303214Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043522.xml
2022-05-18T04:35:23.7724234Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:35:23.7733522Z 
2022-05-18T04:35:23.7733660Z Running tests...
2022-05-18T04:35:23.7734178Z ----------------------------------------------------------------------
2022-05-18T04:35:24.0588478Z   test_reduce_sum_twice (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31671
2022-05-18T04:35:24.0610211Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31672
2022-05-18T04:35:24.0633628Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31673
2022-05-18T04:35:24.8684997Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:35:24.8769130Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:35:24.8769523Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:35:24.8770154Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:35:24.8770691Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:35:24.8786464Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:35:24.8878806Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:35:24.8879358Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:35:24.9798342Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:35:25.2687146Z ok (1.495s)
2022-05-18T04:35:25.2687312Z 
2022-05-18T04:35:25.2687664Z ----------------------------------------------------------------------
2022-05-18T04:35:25.2687999Z Ran 1 test in 1.495s
2022-05-18T04:35:25.2688116Z 
2022-05-18T04:35:25.2688177Z OK
2022-05-18T04:35:25.2688269Z 
2022-05-18T04:35:25.2688363Z Generating XML reports...
2022-05-18T04:35:25.2719196Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043523.xml
2022-05-18T04:35:26.2066108Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:35:26.2076783Z 
2022-05-18T04:35:26.2077066Z Running tests...
2022-05-18T04:35:26.2077487Z ----------------------------------------------------------------------
2022-05-18T04:35:26.4919981Z   test_scatter (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31727
2022-05-18T04:35:26.4942237Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31728
2022-05-18T04:35:26.4965346Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31729
2022-05-18T04:35:27.3187988Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:35:27.3289678Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:35:27.3290085Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:35:27.3290694Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:35:27.3291228Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:35:27.3291757Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:35:27.3397250Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:35:27.4303393Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:35:27.4304131Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:35:27.7019773Z ok (1.494s)
2022-05-18T04:35:27.7020254Z 
2022-05-18T04:35:27.7020777Z ----------------------------------------------------------------------
2022-05-18T04:35:27.7021179Z Ran 1 test in 1.494s
2022-05-18T04:35:27.7021298Z 
2022-05-18T04:35:27.7021360Z OK
2022-05-18T04:35:27.7021452Z 
2022-05-18T04:35:27.7021548Z Generating XML reports...
2022-05-18T04:35:27.7051926Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043526.xml
2022-05-18T04:35:28.6304015Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:35:28.6313745Z 
2022-05-18T04:35:28.6314171Z Running tests...
2022-05-18T04:35:28.6314579Z ----------------------------------------------------------------------
2022-05-18T04:35:28.9201305Z   test_scatter_checks (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31783
2022-05-18T04:35:28.9224007Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31784
2022-05-18T04:35:28.9247527Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31785
2022-05-18T04:35:29.7118639Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:35:29.7133438Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:35:29.7133837Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:35:29.7134456Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:35:29.7134991Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:35:29.7219740Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:35:29.7243196Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:35:29.7243647Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:35:29.8232133Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:35:30.0299319Z ok (1.398s)
2022-05-18T04:35:30.0299499Z 
2022-05-18T04:35:30.0299948Z ----------------------------------------------------------------------
2022-05-18T04:35:30.0300188Z Ran 1 test in 1.398s
2022-05-18T04:35:30.0300303Z 
2022-05-18T04:35:30.0300368Z OK
2022-05-18T04:35:30.0300464Z 
2022-05-18T04:35:30.0300558Z Generating XML reports...
2022-05-18T04:35:30.0332160Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043528.xml
2022-05-18T04:35:30.9751731Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:35:30.9761624Z 
2022-05-18T04:35:30.9761781Z Running tests...
2022-05-18T04:35:30.9762191Z ----------------------------------------------------------------------
2022-05-18T04:35:31.2622578Z   test_scatter_complex (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31836
2022-05-18T04:35:31.2644902Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31837
2022-05-18T04:35:31.2667531Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31838
2022-05-18T04:35:32.0721271Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:35:32.0821875Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:35:32.0822341Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:35:32.0823372Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:35:32.0823891Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:35:32.0824414Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:35:32.0931223Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:35:32.0931915Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:35:32.1835036Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:35:32.4720424Z ok (1.496s)
2022-05-18T04:35:32.4720642Z 
2022-05-18T04:35:32.4721179Z ----------------------------------------------------------------------
2022-05-18T04:35:32.4721557Z Ran 1 test in 1.496s
2022-05-18T04:35:32.4721672Z 
2022-05-18T04:35:32.4721720Z OK
2022-05-18T04:35:32.4721811Z 
2022-05-18T04:35:32.4721910Z Generating XML reports...
2022-05-18T04:35:32.4755300Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043530.xml
2022-05-18T04:35:33.4160754Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:35:33.4171771Z 
2022-05-18T04:35:33.4172220Z Running tests...
2022-05-18T04:35:33.4172866Z ----------------------------------------------------------------------
2022-05-18T04:35:33.4189221Z   test_scatter_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA gather (0.002s)
2022-05-18T04:35:33.4189549Z 
2022-05-18T04:35:33.4189903Z ----------------------------------------------------------------------
2022-05-18T04:35:33.4190262Z Ran 1 test in 0.002s
2022-05-18T04:35:33.4190399Z 
2022-05-18T04:35:33.4190474Z OK (skipped=1)
2022-05-18T04:35:33.4190583Z 
2022-05-18T04:35:33.4190668Z Generating XML reports...
2022-05-18T04:35:33.4224706Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043533.xml
2022-05-18T04:35:34.2563338Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:35:34.2574362Z 
2022-05-18T04:35:34.2574667Z Running tests...
2022-05-18T04:35:34.2575277Z ----------------------------------------------------------------------
2022-05-18T04:35:34.2591586Z   test_scatter_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA gather (0.002s)
2022-05-18T04:35:34.2592244Z 
2022-05-18T04:35:34.2593074Z ----------------------------------------------------------------------
2022-05-18T04:35:34.2593547Z Ran 1 test in 0.002s
2022-05-18T04:35:34.2593674Z 
2022-05-18T04:35:34.2593758Z OK (skipped=1)
2022-05-18T04:35:34.2593892Z 
2022-05-18T04:35:34.2593981Z Generating XML reports...
2022-05-18T04:35:34.2624647Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043534.xml
2022-05-18T04:35:35.1004073Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:35:35.1013858Z 
2022-05-18T04:35:35.1013982Z Running tests...
2022-05-18T04:35:35.1014599Z ----------------------------------------------------------------------
2022-05-18T04:35:35.3855738Z   test_scatter_full_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31912
2022-05-18T04:35:35.3878213Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31913
2022-05-18T04:35:35.3901464Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31914
2022-05-18T04:35:36.2372850Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:35:36.2373611Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:35:36.2374051Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:35:36.2374673Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:35:36.2375203Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:35:36.2375723Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:35:36.2481710Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:35:36.2482664Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:35:36.3385859Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:35:36.3493627Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:35:36.3595817Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:35:36.3596403Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-05-18T04:35:36.3597386Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:35:36.3598183Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:35:36.3598798Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:35:36.5952776Z ok (1.494s)
2022-05-18T04:35:36.5952934Z 
2022-05-18T04:35:36.5953290Z ----------------------------------------------------------------------
2022-05-18T04:35:36.5953574Z Ran 1 test in 1.494s
2022-05-18T04:35:36.5953688Z 
2022-05-18T04:35:36.5953737Z OK
2022-05-18T04:35:36.5953829Z 
2022-05-18T04:35:36.5953931Z Generating XML reports...
2022-05-18T04:35:36.5984477Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043535.xml
2022-05-18T04:35:37.5338802Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:35:37.5348968Z 
2022-05-18T04:35:37.5349099Z Running tests...
2022-05-18T04:35:37.5349709Z ----------------------------------------------------------------------
2022-05-18T04:35:37.8188604Z   test_scatter_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31977
2022-05-18T04:35:37.8211169Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31978
2022-05-18T04:35:37.8234500Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31979
2022-05-18T04:35:38.6511506Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:35:38.6512032Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:35:38.6512483Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:35:38.6513138Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:35:38.6513672Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:35:38.6514182Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:35:38.6523139Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:35:38.6523956Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:35:38.6526393Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:35:38.6526902Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:35:38.6729918Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:35:38.6730412Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-05-18T04:35:38.6731037Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:35:38.6731721Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:35:38.6830171Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-05-18T04:35:39.0288340Z ok (1.494s)
2022-05-18T04:35:39.0288592Z 
2022-05-18T04:35:39.0289164Z ----------------------------------------------------------------------
2022-05-18T04:35:39.0289493Z Ran 1 test in 1.494s
2022-05-18T04:35:39.0289608Z 
2022-05-18T04:35:39.0289668Z OK
2022-05-18T04:35:39.0289759Z 
2022-05-18T04:35:39.0289853Z Generating XML reports...
2022-05-18T04:35:39.0320681Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043537.xml
2022-05-18T04:35:39.9698199Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:35:39.9708669Z 
2022-05-18T04:35:39.9708770Z Running tests...
2022-05-18T04:35:39.9709828Z ----------------------------------------------------------------------
2022-05-18T04:35:40.2588883Z   test_scatter_object_list (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32038
2022-05-18T04:35:40.2611599Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32039
2022-05-18T04:35:40.2635351Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 32040
2022-05-18T04:35:41.1035639Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:35:41.1134475Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:35:41.1135085Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:35:41.1136024Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:35:41.1136806Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:35:41.1137680Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:35:41.1244161Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:35:41.1244728Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:35:41.2150665Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:35:41.4689318Z ok (1.498s)
2022-05-18T04:35:41.4689580Z 
2022-05-18T04:35:41.4690094Z ----------------------------------------------------------------------
2022-05-18T04:35:41.4690364Z Ran 1 test in 1.498s
2022-05-18T04:35:41.4690479Z 
2022-05-18T04:35:41.4690541Z OK
2022-05-18T04:35:41.4690633Z 
2022-05-18T04:35:41.4690712Z Generating XML reports...
2022-05-18T04:35:41.4721519Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043539.xml
2022-05-18T04:35:42.4240569Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:35:42.4250349Z 
2022-05-18T04:35:42.4250481Z Running tests...
2022-05-18T04:35:42.4250981Z ----------------------------------------------------------------------
2022-05-18T04:35:42.7153083Z   test_send_recv (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32091
2022-05-18T04:35:42.7175758Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32092
2022-05-18T04:35:42.7198574Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 32093
2022-05-18T04:35:43.5142757Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:35:43.5243793Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:35:43.5244444Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:35:43.5245088Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:35:43.5245623Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:35:43.5246146Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:35:43.5352204Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:35:43.6258131Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:35:43.6258735Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:35:43.8249081Z ok (1.400s)
2022-05-18T04:35:43.8249318Z 
2022-05-18T04:35:43.8249768Z ----------------------------------------------------------------------
2022-05-18T04:35:43.8250162Z Ran 1 test in 1.400s
2022-05-18T04:35:43.8250349Z 
2022-05-18T04:35:43.8250436Z OK
2022-05-18T04:35:43.8250581Z 
2022-05-18T04:35:43.8250726Z Generating XML reports...
2022-05-18T04:35:43.8282704Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043542.xml
2022-05-18T04:35:44.7616647Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:35:44.7626670Z 
2022-05-18T04:35:44.7626987Z Running tests...
2022-05-18T04:35:44.7627632Z ----------------------------------------------------------------------
2022-05-18T04:35:45.0464673Z   test_send_recv_any_source (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32144
2022-05-18T04:35:45.0486198Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32145
2022-05-18T04:35:45.0508937Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 32146
2022-05-18T04:35:45.8392494Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:35:45.8405660Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:35:45.8406073Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:35:45.8406696Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:35:45.8407214Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:35:45.8493773Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:35:45.8515080Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:35:45.8516180Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:35:45.9507132Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:35:46.1560570Z ok (1.393s)
2022-05-18T04:35:46.1560988Z 
2022-05-18T04:35:46.1561524Z ----------------------------------------------------------------------
2022-05-18T04:35:46.1561862Z Ran 1 test in 1.393s
2022-05-18T04:35:46.1561976Z 
2022-05-18T04:35:46.1562029Z OK
2022-05-18T04:35:46.1562121Z 
2022-05-18T04:35:46.1562215Z Generating XML reports...
2022-05-18T04:35:46.1592924Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043544.xml
2022-05-18T04:35:47.1207147Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:35:47.1216620Z 
2022-05-18T04:35:47.1216757Z Running tests...
2022-05-18T04:35:47.1217332Z ----------------------------------------------------------------------
2022-05-18T04:35:47.4085883Z   test_send_recv_any_source_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32197
2022-05-18T04:35:47.4107957Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32198
2022-05-18T04:35:47.4130714Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 32199
2022-05-18T04:35:48.2258727Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:35:48.2346323Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:35:48.2346975Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:35:48.2347591Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:35:48.2348127Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:35:48.2359885Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:35:48.2455613Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:35:48.2456581Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:35:48.3371909Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:35:48.6184530Z ok (1.496s)
2022-05-18T04:35:48.6184746Z 
2022-05-18T04:35:48.6185182Z ----------------------------------------------------------------------
2022-05-18T04:35:48.6185489Z Ran 1 test in 1.497s
2022-05-18T04:35:48.6185610Z 
2022-05-18T04:35:48.6185659Z OK
2022-05-18T04:35:48.6185750Z 
2022-05-18T04:35:48.6185847Z Generating XML reports...
2022-05-18T04:35:48.6216707Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043547.xml
2022-05-18T04:35:49.5660711Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:35:49.5671282Z 
2022-05-18T04:35:49.5671422Z Running tests...
2022-05-18T04:35:49.5672118Z ----------------------------------------------------------------------
2022-05-18T04:35:49.8526007Z   test_send_recv_any_source_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32253
2022-05-18T04:35:49.8548231Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32254
2022-05-18T04:35:49.8570930Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 32255
2022-05-18T04:35:50.6571271Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:35:50.6672125Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:35:50.6673006Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:35:50.6673699Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:35:50.6674237Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:35:50.6674761Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:35:50.6682305Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:35:50.6683043Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:35:50.6683718Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:35:50.9621722Z ok (1.395s)
2022-05-18T04:35:50.9621937Z 
2022-05-18T04:35:50.9622326Z ----------------------------------------------------------------------
2022-05-18T04:35:50.9622627Z Ran 1 test in 1.395s
2022-05-18T04:35:50.9622729Z 
2022-05-18T04:35:50.9622791Z OK
2022-05-18T04:35:50.9623009Z 
2022-05-18T04:35:50.9623134Z Generating XML reports...
2022-05-18T04:35:50.9654907Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043549.xml
2022-05-18T04:35:51.8983165Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:35:51.8992848Z 
2022-05-18T04:35:51.8992947Z Running tests...
2022-05-18T04:35:51.8993913Z ----------------------------------------------------------------------
2022-05-18T04:35:52.1844707Z   test_send_recv_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32309
2022-05-18T04:35:52.1867589Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32310
2022-05-18T04:35:52.1889902Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 32311
2022-05-18T04:35:53.0236228Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:35:53.0335680Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:35:53.0336388Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:35:53.0337405Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:35:53.0338073Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:35:53.0338600Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:35:53.0443765Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:35:53.1351133Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:35:53.1351675Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:35:53.3943134Z ok (1.495s)
2022-05-18T04:35:53.3943373Z 
2022-05-18T04:35:53.3943904Z ----------------------------------------------------------------------
2022-05-18T04:35:53.3944233Z Ran 1 test in 1.495s
2022-05-18T04:35:53.3944353Z 
2022-05-18T04:35:53.3944401Z OK
2022-05-18T04:35:53.3944491Z 
2022-05-18T04:35:53.3944587Z Generating XML reports...
2022-05-18T04:35:53.3974806Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043551.xml
2022-05-18T04:35:54.3317284Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:35:54.3327278Z 
2022-05-18T04:35:54.3327407Z Running tests...
2022-05-18T04:35:54.3328055Z ----------------------------------------------------------------------
2022-05-18T04:35:54.3343125Z   test_send_recv_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Send Recv Only (0.001s)
2022-05-18T04:35:54.3343665Z 
2022-05-18T04:35:54.3343939Z ----------------------------------------------------------------------
2022-05-18T04:35:54.3344193Z Ran 1 test in 0.002s
2022-05-18T04:35:54.3344295Z 
2022-05-18T04:35:54.3344373Z OK (skipped=1)
2022-05-18T04:35:54.3344480Z 
2022-05-18T04:35:54.3344567Z Generating XML reports...
2022-05-18T04:35:54.3375335Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043554.xml
2022-05-18T04:35:55.1773072Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:35:55.1783057Z 
2022-05-18T04:35:55.1783422Z Running tests...
2022-05-18T04:35:55.1784031Z ----------------------------------------------------------------------
2022-05-18T04:35:55.1799387Z   test_send_recv_nccl_autograd_profiler (__main__.TestDistBackendWithSpawn) ... skip: NCCL Send Recv Only (0.002s)
2022-05-18T04:35:55.1799808Z 
2022-05-18T04:35:55.1800299Z ----------------------------------------------------------------------
2022-05-18T04:35:55.1800650Z Ran 1 test in 0.002s
2022-05-18T04:35:55.1800767Z 
2022-05-18T04:35:55.1800839Z OK (skipped=1)
2022-05-18T04:35:55.1800947Z 
2022-05-18T04:35:55.1801019Z Generating XML reports...
2022-05-18T04:35:55.1831982Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043555.xml
2022-05-18T04:35:56.0363697Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:35:56.0373934Z 
2022-05-18T04:35:56.0374070Z Running tests...
2022-05-18T04:35:56.0374663Z ----------------------------------------------------------------------
2022-05-18T04:35:56.0392331Z   test_send_recv_nccl_torch_profiler (__main__.TestDistBackendWithSpawn) ... skip: NCCL Send Recv Only (0.002s)
2022-05-18T04:35:56.0392583Z 
2022-05-18T04:35:56.0392897Z ----------------------------------------------------------------------
2022-05-18T04:35:56.0393145Z Ran 1 test in 0.002s
2022-05-18T04:35:56.0393270Z 
2022-05-18T04:35:56.0393348Z OK (skipped=1)
2022-05-18T04:35:56.0393460Z 
2022-05-18T04:35:56.0393532Z Generating XML reports...
2022-05-18T04:35:56.0427102Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043556.xml
2022-05-18T04:35:56.8882130Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:35:56.8891760Z 
2022-05-18T04:35:56.8892083Z Running tests...
2022-05-18T04:35:56.8892671Z ----------------------------------------------------------------------
2022-05-18T04:35:57.1730737Z   test_send_recv_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32395
2022-05-18T04:35:57.1753290Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32396
2022-05-18T04:35:57.1776164Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 32397
2022-05-18T04:35:57.9904415Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:35:57.9905080Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:35:57.9905661Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:35:57.9906643Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:35:57.9907276Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:35:57.9907788Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:35:58.0012175Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:35:58.0012899Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:35:58.0916271Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:35:58.3831303Z ok (1.494s)
2022-05-18T04:35:58.3831579Z 
2022-05-18T04:35:58.3832045Z ----------------------------------------------------------------------
2022-05-18T04:35:58.3832296Z Ran 1 test in 1.494s
2022-05-18T04:35:58.3832413Z 
2022-05-18T04:35:58.3832473Z OK
2022-05-18T04:35:58.3832564Z 
2022-05-18T04:35:58.3832662Z Generating XML reports...
2022-05-18T04:35:58.3868765Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043556.xml
2022-05-18T04:35:59.3220840Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:35:59.3230566Z 
2022-05-18T04:35:59.3230753Z Running tests...
2022-05-18T04:35:59.3231236Z ----------------------------------------------------------------------
2022-05-18T04:35:59.6096780Z   test_send_recv_with_tag (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32451
2022-05-18T04:35:59.6118905Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32452
2022-05-18T04:35:59.6142162Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 32453
2022-05-18T04:36:00.4311525Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:36:00.4312275Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:36:00.4312937Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:36:00.4313790Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:36:00.4314352Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:36:00.4314863Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:36:00.4323573Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:36:00.4324048Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:36:00.5325817Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:36:00.7193403Z ok (1.396s)
2022-05-18T04:36:00.7193715Z 
2022-05-18T04:36:00.7194260Z ----------------------------------------------------------------------
2022-05-18T04:36:00.7194535Z Ran 1 test in 1.396s
2022-05-18T04:36:00.7194651Z 
2022-05-18T04:36:00.7194730Z OK
2022-05-18T04:36:00.7194822Z 
2022-05-18T04:36:00.7194901Z Generating XML reports...
2022-05-18T04:36:00.7224883Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043559.xml
2022-05-18T04:36:01.6684926Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:36:01.6694780Z 
2022-05-18T04:36:01.6694937Z Running tests...
2022-05-18T04:36:01.6695356Z ----------------------------------------------------------------------
2022-05-18T04:36:01.9567993Z   test_send_recv_with_tag_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32504
2022-05-18T04:36:01.9589206Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32505
2022-05-18T04:36:01.9611801Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 32506
2022-05-18T04:36:02.7863407Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:36:02.7864333Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:36:02.7864989Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:36:02.7865971Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:36:02.7866749Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:36:02.7867282Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:36:02.7970280Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:36:02.8879373Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:36:02.8879963Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:36:03.0663226Z ok (1.396s)
2022-05-18T04:36:03.0663513Z 
2022-05-18T04:36:03.0664010Z ----------------------------------------------------------------------
2022-05-18T04:36:03.0664266Z Ran 1 test in 1.397s
2022-05-18T04:36:03.0664381Z 
2022-05-18T04:36:03.0664443Z OK
2022-05-18T04:36:03.0664534Z 
2022-05-18T04:36:03.0664629Z Generating XML reports...
2022-05-18T04:36:03.0694759Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043601.xml
2022-05-18T04:36:04.0069311Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:36:04.0079174Z 
2022-05-18T04:36:04.0079269Z Running tests...
2022-05-18T04:36:04.0079728Z ----------------------------------------------------------------------
2022-05-18T04:36:04.2946532Z   test_send_recv_with_tag_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32560
2022-05-18T04:36:04.2968094Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32561
2022-05-18T04:36:04.2992245Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 32562
2022-05-18T04:36:05.0668369Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:36:05.0769908Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:36:05.0770609Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:36:05.0771243Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:36:05.0771759Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:36:05.0772295Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:36:05.0790030Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:36:05.0790539Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:36:05.0790899Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:36:05.4044227Z ok (1.396s)
2022-05-18T04:36:05.4044477Z 
2022-05-18T04:36:05.4044993Z ----------------------------------------------------------------------
2022-05-18T04:36:05.4045299Z Ran 1 test in 1.396s
2022-05-18T04:36:05.4045414Z 
2022-05-18T04:36:05.4045792Z OK
2022-05-18T04:36:05.4045925Z 
2022-05-18T04:36:05.4046022Z Generating XML reports...
2022-05-18T04:36:05.4076868Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043604.xml
2022-05-18T04:36:06.3465678Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:36:06.3475502Z 
2022-05-18T04:36:06.3475606Z Running tests...
2022-05-18T04:36:06.3476503Z ----------------------------------------------------------------------
2022-05-18T04:36:06.6325779Z   test_sparse_all_reduce_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32616
2022-05-18T04:36:06.6347632Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32617
2022-05-18T04:36:06.6370843Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 32618
2022-05-18T04:36:07.5137207Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:36:07.5137841Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:36:07.5138211Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:36:07.5138838Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:36:07.5139374Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:36:07.5139894Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:36:07.5246062Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:36:07.6151539Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:36:07.6151942Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:36:07.8424186Z ok (1.494s)
2022-05-18T04:36:07.8424356Z 
2022-05-18T04:36:07.8424744Z ----------------------------------------------------------------------
2022-05-18T04:36:07.8425024Z Ran 1 test in 1.495s
2022-05-18T04:36:07.8425138Z 
2022-05-18T04:36:07.8425199Z OK
2022-05-18T04:36:07.8425289Z 
2022-05-18T04:36:07.8425367Z Generating XML reports...
2022-05-18T04:36:07.8457838Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043606.xml
2022-05-18T04:36:08.7827380Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:36:08.7837639Z 
2022-05-18T04:36:08.7837736Z Running tests...
2022-05-18T04:36:08.7838537Z ----------------------------------------------------------------------
2022-05-18T04:36:09.0680063Z   test_sparse_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32696
2022-05-18T04:36:09.0701430Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32697
2022-05-18T04:36:09.0724680Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 32698
2022-05-18T04:36:09.8559134Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:36:09.8597213Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:36:09.8597649Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:36:09.8598272Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:36:09.8598793Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:36:09.8660413Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:36:09.8707565Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:36:09.8708203Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:36:09.9674227Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:36:10.1774241Z skip: CUDA is not available. (1.393s)
2022-05-18T04:36:10.1774498Z 
2022-05-18T04:36:10.1774949Z ----------------------------------------------------------------------
2022-05-18T04:36:10.1775385Z Ran 1 test in 1.394s
2022-05-18T04:36:10.1775562Z 
2022-05-18T04:36:10.1775659Z OK (skipped=1)
2022-05-18T04:36:10.1775833Z 
2022-05-18T04:36:10.1775970Z Generating XML reports...
2022-05-18T04:36:10.1807800Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043608.xml
2022-05-18T04:36:11.1193741Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:36:11.1203339Z 
2022-05-18T04:36:11.1203488Z Running tests...
2022-05-18T04:36:11.1204110Z ----------------------------------------------------------------------
2022-05-18T04:36:11.4075487Z   test_stateless_api_with_ddp (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32749
2022-05-18T04:36:11.4097656Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32750
2022-05-18T04:36:11.4120688Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 32751
2022-05-18T04:36:12.2035890Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:36:12.2136067Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:36:12.2136778Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:36:12.2137454Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:36:12.2138010Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:36:12.2138534Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:36:12.2244245Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:36:12.3151047Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:36:12.3151667Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:36:12.5169957Z skip: Need at least 2 CUDA devices (1.396s)
2022-05-18T04:36:12.5170236Z 
2022-05-18T04:36:12.5170569Z ----------------------------------------------------------------------
2022-05-18T04:36:12.5170811Z Ran 1 test in 1.397s
2022-05-18T04:36:12.5170925Z 
2022-05-18T04:36:12.5171003Z OK (skipped=1)
2022-05-18T04:36:12.5171113Z 
2022-05-18T04:36:12.5171215Z Generating XML reports...
2022-05-18T04:36:12.5202604Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043611.xml
2022-05-18T04:36:13.4538039Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:36:13.4548850Z 
2022-05-18T04:36:13.4549312Z Running tests...
2022-05-18T04:36:13.4549793Z ----------------------------------------------------------------------
2022-05-18T04:36:13.7390816Z   test_static_graph_api_cpu (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 334
2022-05-18T04:36:13.7413085Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 335
2022-05-18T04:36:13.7437314Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 336
2022-05-18T04:36:14.6009165Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:36:14.6110000Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:36:14.6110585Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:36:14.6111215Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:36:14.6111731Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:36:14.6112253Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:36:14.6119662Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:36:14.6120861Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:36:14.6122007Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:36:14.6184238Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5z14ljdk
2022-05-18T04:36:14.6185444Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1ppfoghu
2022-05-18T04:36:14.6186153Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5z14ljdk/_remote_module_non_scriptable.py
2022-05-18T04:36:14.6186750Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwm9qdch1
2022-05-18T04:36:14.6187657Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1ppfoghu/_remote_module_non_scriptable.py
2022-05-18T04:36:14.6189478Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwm9qdch1/_remote_module_non_scriptable.py
2022-05-18T04:36:14.8488805Z ok (1.394s)
2022-05-18T04:36:14.8489024Z 
2022-05-18T04:36:14.8489535Z ----------------------------------------------------------------------
2022-05-18T04:36:14.8489987Z Ran 1 test in 1.394s
2022-05-18T04:36:14.8490121Z 
2022-05-18T04:36:14.8490170Z OK
2022-05-18T04:36:14.8490262Z 
2022-05-18T04:36:14.8490354Z Generating XML reports...
2022-05-18T04:36:14.8521548Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043613.xml
2022-05-18T04:36:15.7872741Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:36:15.7882557Z 
2022-05-18T04:36:15.7882922Z Running tests...
2022-05-18T04:36:15.7883346Z ----------------------------------------------------------------------
2022-05-18T04:36:16.0737029Z   test_sync_bn_logged (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 387
2022-05-18T04:36:16.0758626Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 388
2022-05-18T04:36:16.0782166Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 389
2022-05-18T04:36:16.9125533Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:36:16.9227305Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:36:16.9227986Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:36:16.9228394Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:36:16.9229135Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:36:16.9230080Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:36:16.9238340Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:36:16.9238834Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:36:16.9239842Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:36:17.0832327Z skip: Need at least 2 CUDA devices (1.295s)
2022-05-18T04:36:17.0832845Z 
2022-05-18T04:36:17.0833305Z ----------------------------------------------------------------------
2022-05-18T04:36:17.0833695Z Ran 1 test in 1.295s
2022-05-18T04:36:17.0833886Z 
2022-05-18T04:36:17.0833998Z OK (skipped=1)
2022-05-18T04:36:17.0834157Z 
2022-05-18T04:36:17.0834306Z Generating XML reports...
2022-05-18T04:36:17.0866942Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043615.xml
2022-05-18T04:36:18.0296292Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:36:18.0306928Z 
2022-05-18T04:36:18.0307230Z Running tests...
2022-05-18T04:36:18.0307857Z ----------------------------------------------------------------------
2022-05-18T04:36:18.3172266Z   test_undefined_grad_parity_unused_parameters (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 440
2022-05-18T04:36:18.3195256Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 441
2022-05-18T04:36:18.3217720Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 442
2022-05-18T04:36:19.1547661Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:36:19.1648961Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:36:19.1649546Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:36:19.1650462Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:36:19.1651324Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:36:19.1652201Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-05-18T04:36:19.1661469Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:36:19.1662016Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:36:19.1663229Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:36:19.3266455Z skip: Need at least 2 CUDA devices (1.296s)
2022-05-18T04:36:19.3266657Z 
2022-05-18T04:36:19.3267008Z ----------------------------------------------------------------------
2022-05-18T04:36:19.3267420Z Ran 1 test in 1.296s
2022-05-18T04:36:19.3267632Z 
2022-05-18T04:36:19.3267732Z OK (skipped=1)
2022-05-18T04:36:19.3267935Z 
2022-05-18T04:36:19.3268069Z Generating XML reports...
2022-05-18T04:36:19.3299867Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043618.xml
2022-05-18T04:36:20.2659793Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:36:20.2670749Z 
2022-05-18T04:36:20.2671060Z Running tests...
2022-05-18T04:36:20.2671693Z ----------------------------------------------------------------------
2022-05-18T04:36:20.2686660Z   test_verify_model_across_rank_with_logger (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'nccl', 'gloo'} (0.001s)
2022-05-18T04:36:20.2687130Z 
2022-05-18T04:36:20.2687666Z ----------------------------------------------------------------------
2022-05-18T04:36:20.2688129Z Ran 1 test in 0.002s
2022-05-18T04:36:20.2688340Z 
2022-05-18T04:36:20.2688471Z OK (skipped=1)
2022-05-18T04:36:20.2688624Z 
2022-05-18T04:36:20.2688700Z Generating XML reports...
2022-05-18T04:36:20.2719320Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043620.xml
2022-05-18T04:36:21.1135009Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-05-18T04:36:21.1145875Z 
2022-05-18T04:36:21.1146468Z Running tests...
2022-05-18T04:36:21.1147126Z ----------------------------------------------------------------------
2022-05-18T04:36:21.1162434Z   test_verify_model_across_rank_without_logger (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'nccl'} (0.001s)
2022-05-18T04:36:21.1162902Z 
2022-05-18T04:36:21.1163250Z ----------------------------------------------------------------------
2022-05-18T04:36:21.1163655Z Ran 1 test in 0.002s
2022-05-18T04:36:21.1163839Z 
2022-05-18T04:36:21.1163944Z OK (skipped=1)
2022-05-18T04:36:21.1164114Z 
2022-05-18T04:36:21.1164255Z Generating XML reports...
2022-05-18T04:36:21.1196414Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043621.xml
2022-05-18T04:36:21.3452151Z Running distributed/test_launcher ... [2022-05-18 04:36:21.344890]
2022-05-18T04:36:21.3452741Z Executing ['/opt/conda/bin/python', 'distributed/test_launcher.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-05-18 04:36:21.344962]
2022-05-18T04:36:22.0798716Z Test results will be stored in test-reports/python-unittest/distributed.test_launcher
2022-05-18T04:36:22.0808868Z 
2022-05-18T04:36:22.0808972Z Running tests...
2022-05-18T04:36:22.0809446Z ----------------------------------------------------------------------
2022-05-18T04:36:22.3615630Z   test_launch_user_script (__main__.TestDistributedLaunch) ... /opt/conda/lib/python3.7/site-packages/torch/distributed/launch.py:186: FutureWarning: The module torch.distributed.launch is deprecated
2022-05-18T04:36:22.3616119Z and will be removed in future. Use torchrun.
2022-05-18T04:36:22.3616432Z Note that --use_env is set by default in torchrun.
2022-05-18T04:36:22.3616812Z If your script expects `--local_rank` argument to be set, please
2022-05-18T04:36:22.3617167Z change it to read from `os.environ['LOCAL_RANK']` instead. See 
2022-05-18T04:36:22.3617601Z https://pytorch.org/docs/stable/distributed.html#launch-utility for 
2022-05-18T04:36:22.3617862Z further instructions
2022-05-18T04:36:22.3617966Z 
2022-05-18T04:36:22.3618040Z   FutureWarning,
2022-05-18T04:36:22.3625837Z WARNING:torch.distributed.run:
2022-05-18T04:36:22.3626129Z *****************************************
2022-05-18T04:36:22.3626525Z Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. 
2022-05-18T04:36:22.3626966Z *****************************************
2022-05-18T04:36:22.3898928Z Success, smoke test
2022-05-18T04:36:22.3900251Z Success, smoke test
2022-05-18T04:36:22.3929452Z Success, smoke test
2022-05-18T04:36:22.3934349Z Success, smoke test
2022-05-18T04:36:23.3827793Z ok (1.302s)
2022-05-18T04:36:23.3828077Z 
2022-05-18T04:36:23.3828525Z ----------------------------------------------------------------------
2022-05-18T04:36:23.3828936Z Ran 1 test in 1.302s
2022-05-18T04:36:23.3829124Z 
2022-05-18T04:36:23.3829227Z OK
2022-05-18T04:36:23.3829357Z 
2022-05-18T04:36:23.3829563Z Generating XML reports...
2022-05-18T04:36:23.3872705Z Generated XML report: test-reports/python-unittest/distributed.test_launcher/TEST-TestDistributedLaunch-20220518043622.xml
2022-05-18T04:36:23.5894433Z Running distributed/test_nccl ... [2022-05-18 04:36:23.589044]
2022-05-18T04:36:23.5894975Z Executing ['/opt/conda/bin/python', 'distributed/test_nccl.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-05-18 04:36:23.589124]
2022-05-18T04:36:24.4439518Z CUDA not available, skipping tests
2022-05-18T04:36:24.4458674Z Test results will be stored in test-reports/python-unittest/distributed.test_nccl
2022-05-18T04:36:24.4470090Z 
2022-05-18T04:36:24.4470320Z Running tests...
2022-05-18T04:36:24.4471282Z ----------------------------------------------------------------------
2022-05-18T04:36:24.4471571Z 
2022-05-18T04:36:24.4471825Z ----------------------------------------------------------------------
2022-05-18T04:36:24.4472159Z Ran 0 tests in 0.000s
2022-05-18T04:36:24.4472276Z 
2022-05-18T04:36:24.4472324Z OK
2022-05-18T04:36:24.4472418Z 
2022-05-18T04:36:24.4472513Z Generating XML reports...
2022-05-18T04:36:24.6173783Z Running distributed/test_pg_wrapper ... [2022-05-18 04:36:24.617028]
2022-05-18T04:36:24.6174364Z Executing ['/opt/conda/bin/python', 'distributed/test_pg_wrapper.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-05-18 04:36:24.617110]
2022-05-18T04:36:25.1832767Z <unittest.suite.TestSuite tests=[]>
2022-05-18T04:36:25.1833286Z <unittest.suite.TestSuite tests=[]>
2022-05-18T04:36:25.1835063Z <unittest.suite.TestSuite tests=[<__main__.ProcessGroupGlooWrapperTest testMethod=test_collective_hang>, <__main__.ProcessGroupGlooWrapperTest testMethod=test_collective_shape_mismatch>, <__main__.ProcessGroupGlooWrapperTest testMethod=test_collective_shape_mismatch_cuda>, <__main__.ProcessGroupGlooWrapperTest testMethod=test_collective_shape_mismatch_cuda_debug_mode>, <__main__.ProcessGroupGlooWrapperTest testMethod=test_collective_shape_mismatch_debug_mode>, <__main__.ProcessGroupGlooWrapperTest testMethod=test_collectives_op_mismatch>, <__main__.ProcessGroupGlooWrapperTest testMethod=test_collectives_op_mismatch_cuda>, <__main__.ProcessGroupGlooWrapperTest testMethod=test_collectives_op_mismatch_cuda_debug_mode>, <__main__.ProcessGroupGlooWrapperTest testMethod=test_collectives_op_mismatch_debug_mode>]>
2022-05-18T04:36:25.1836850Z test_collective_hang (__main__.ProcessGroupGlooWrapperTest)
2022-05-18T04:36:25.1837173Z test_collective_shape_mismatch (__main__.ProcessGroupGlooWrapperTest)
2022-05-18T04:36:25.1837517Z test_collective_shape_mismatch_cuda (__main__.ProcessGroupGlooWrapperTest)
2022-05-18T04:36:25.1837874Z test_collective_shape_mismatch_cuda_debug_mode (__main__.ProcessGroupGlooWrapperTest)
2022-05-18T04:36:25.1838239Z test_collective_shape_mismatch_debug_mode (__main__.ProcessGroupGlooWrapperTest)
2022-05-18T04:36:25.1838564Z test_collectives_op_mismatch (__main__.ProcessGroupGlooWrapperTest)
2022-05-18T04:36:25.1838894Z test_collectives_op_mismatch_cuda (__main__.ProcessGroupGlooWrapperTest)
2022-05-18T04:36:25.1839237Z test_collectives_op_mismatch_cuda_debug_mode (__main__.ProcessGroupGlooWrapperTest)
2022-05-18T04:36:25.1839588Z test_collectives_op_mismatch_debug_mode (__main__.ProcessGroupGlooWrapperTest)
2022-05-18T04:36:25.1840359Z <unittest.suite.TestSuite tests=[<__main__.ProcessGroupNCCLWrapperTest testMethod=test_collective_hang>, <__main__.ProcessGroupNCCLWrapperTest testMethod=test_collective_shape_mismatch>, <__main__.ProcessGroupNCCLWrapperTest testMethod=test_collective_shape_mismatch_debug_mode>, <__main__.ProcessGroupNCCLWrapperTest testMethod=test_collectives_op_mismatch>, <__main__.ProcessGroupNCCLWrapperTest testMethod=test_collectives_op_mismatch_debug_mode>]>
2022-05-18T04:36:25.1841099Z test_collective_hang (__main__.ProcessGroupNCCLWrapperTest)
2022-05-18T04:36:25.1841416Z test_collective_shape_mismatch (__main__.ProcessGroupNCCLWrapperTest)
2022-05-18T04:36:25.1841757Z test_collective_shape_mismatch_debug_mode (__main__.ProcessGroupNCCLWrapperTest)
2022-05-18T04:36:25.1842089Z test_collectives_op_mismatch (__main__.ProcessGroupNCCLWrapperTest)
2022-05-18T04:36:25.1842414Z test_collectives_op_mismatch_debug_mode (__main__.ProcessGroupNCCLWrapperTest)
2022-05-18T04:36:25.7474352Z Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper
2022-05-18T04:36:25.7484058Z 
2022-05-18T04:36:25.7484197Z Running tests...
2022-05-18T04:36:25.7485282Z ----------------------------------------------------------------------
2022-05-18T04:36:26.0305587Z   test_collective_hang (__main__.ProcessGroupGlooWrapperTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 549
2022-05-18T04:36:26.0327786Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 550
2022-05-18T04:36:26.0350055Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 551
2022-05-18T04:36:26.0373340Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 552
2022-05-18T04:36:26.7043255Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:36:26.7045218Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:36:26.7163772Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:36:26.7682685Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:36:26.7874071Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:36:26.7992081Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:36:26.8093454Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T04:36:26.8094447Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:36:26.8095088Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:36:26.8095608Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:36:26.8096135Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:36:26.8178463Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:36:26.9021011Z [E ProcessGroupGloo.cpp:2791] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 2000 ms
2022-05-18T04:36:26.9021430Z [E ProcessGroupGloo.cpp:136] [Rank 0]: Ranks 1 failed to pass monitoredBarrier in 2000 ms
2022-05-18T04:36:26.9121394Z [E ProcessGroupGloo.cpp:136] Rank 2 successfully reached monitoredBarrier, but received errors while waiting for send/recv from rank 0. Please check rank 0 logs for faulty rank.
2022-05-18T04:36:26.9222013Z [E ProcessGroupGloo.cpp:136] Rank 3 successfully reached monitoredBarrier, but received errors while waiting for send/recv from rank 0. Please check rank 0 logs for faulty rank.
2022-05-18T04:36:27.1407104Z ok (1.392s)
2022-05-18T04:36:27.1407326Z 
2022-05-18T04:36:27.1407770Z ----------------------------------------------------------------------
2022-05-18T04:36:27.1408194Z Ran 1 test in 1.392s
2022-05-18T04:36:27.1408357Z 
2022-05-18T04:36:27.1408447Z OK
2022-05-18T04:36:27.1408590Z 
2022-05-18T04:36:27.1408730Z Generating XML reports...
2022-05-18T04:36:27.1442863Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20220518043625.xml
2022-05-18T04:36:27.9083235Z Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper
2022-05-18T04:36:27.9093391Z 
2022-05-18T04:36:27.9093519Z Running tests...
2022-05-18T04:36:27.9094132Z ----------------------------------------------------------------------
2022-05-18T04:36:28.1961380Z   test_collective_shape_mismatch (__main__.ProcessGroupGlooWrapperTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 640
2022-05-18T04:36:28.1983275Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 641
2022-05-18T04:36:28.2005561Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 642
2022-05-18T04:36:28.2029390Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 643
2022-05-18T04:36:28.8274641Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:36:28.8390143Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:36:28.8550692Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:36:28.8689088Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:36:28.8999851Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:36:28.9099716Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:36:28.9202292Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T04:36:28.9202965Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:36:28.9203920Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:36:28.9204851Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:36:28.9205383Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:36:28.9205905Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:36:29.3064145Z ok (1.397s)
2022-05-18T04:36:29.3064359Z 
2022-05-18T04:36:29.3064905Z ----------------------------------------------------------------------
2022-05-18T04:36:29.3065273Z Ran 1 test in 1.397s
2022-05-18T04:36:29.3065386Z 
2022-05-18T04:36:29.3065447Z OK
2022-05-18T04:36:29.3065537Z 
2022-05-18T04:36:29.3065629Z Generating XML reports...
2022-05-18T04:36:29.3099289Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20220518043627.xml
2022-05-18T04:36:30.0699842Z Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper
2022-05-18T04:36:30.0710071Z 
2022-05-18T04:36:30.0710167Z Running tests...
2022-05-18T04:36:30.0710956Z ----------------------------------------------------------------------
2022-05-18T04:36:30.3536850Z   test_collective_shape_mismatch_cuda (__main__.ProcessGroupGlooWrapperTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 731
2022-05-18T04:36:30.3559137Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 732
2022-05-18T04:36:30.3581307Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 733
2022-05-18T04:36:30.3605602Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 734
2022-05-18T04:36:31.0115352Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:36:31.0139636Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:36:31.0140159Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:36:31.0226879Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:36:31.1634056Z skip: Need at least 4 CUDA devices (1.092s)
2022-05-18T04:36:31.1634388Z 
2022-05-18T04:36:31.1634881Z ----------------------------------------------------------------------
2022-05-18T04:36:31.1635135Z Ran 1 test in 1.092s
2022-05-18T04:36:31.1635250Z 
2022-05-18T04:36:31.1635324Z OK (skipped=1)
2022-05-18T04:36:31.1635430Z 
2022-05-18T04:36:31.1635515Z Generating XML reports...
2022-05-18T04:36:31.1668169Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20220518043630.xml
2022-05-18T04:36:31.9223963Z Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper
2022-05-18T04:36:31.9233219Z 
2022-05-18T04:36:31.9233319Z Running tests...
2022-05-18T04:36:31.9233931Z ----------------------------------------------------------------------
2022-05-18T04:36:32.2062991Z   test_collective_shape_mismatch_cuda_debug_mode (__main__.ProcessGroupGlooWrapperTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 786
2022-05-18T04:36:32.2085320Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 787
2022-05-18T04:36:32.2106905Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 788
2022-05-18T04:36:32.2130076Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 789
2022-05-18T04:36:32.8362768Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:36:32.8479416Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:36:32.8606644Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:36:32.8689561Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:36:33.0159006Z skip: Need at least 4 CUDA devices (1.092s)
2022-05-18T04:36:33.0159326Z 
2022-05-18T04:36:33.0159832Z ----------------------------------------------------------------------
2022-05-18T04:36:33.0160091Z Ran 1 test in 1.092s
2022-05-18T04:36:33.0160214Z 
2022-05-18T04:36:33.0160287Z OK (skipped=1)
2022-05-18T04:36:33.0160394Z 
2022-05-18T04:36:33.0160479Z Generating XML reports...
2022-05-18T04:36:33.0193915Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20220518043631.xml
2022-05-18T04:36:33.7743665Z Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper
2022-05-18T04:36:33.7753444Z 
2022-05-18T04:36:33.7753675Z Running tests...
2022-05-18T04:36:33.7754082Z ----------------------------------------------------------------------
2022-05-18T04:36:34.0578078Z   test_collective_shape_mismatch_debug_mode (__main__.ProcessGroupGlooWrapperTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 841
2022-05-18T04:36:34.0598779Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 842
2022-05-18T04:36:34.0621250Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 843
2022-05-18T04:36:34.0644679Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 844
2022-05-18T04:36:34.6596901Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:36:34.6941503Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:36:34.6942181Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:36:34.7016270Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:36:34.7732621Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:36:34.7833077Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:36:34.7935186Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T04:36:34.7935850Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:36:34.7936805Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:36:34.7937373Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:36:34.7937882Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:36:34.7938403Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:36:34.8448205Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:36:34.8650219Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-05-18T04:36:34.8650997Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:36:34.8652162Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
2022-05-18T04:36:34.8652762Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 3
2022-05-18T04:36:34.8653549Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
2022-05-18T04:36:34.8654342Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
2022-05-18T04:36:34.8655185Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
2022-05-18T04:36:35.1678697Z ok (1.392s)
2022-05-18T04:36:35.1679003Z 
2022-05-18T04:36:35.1679529Z ----------------------------------------------------------------------
2022-05-18T04:36:35.1679985Z Ran 1 test in 1.392s
2022-05-18T04:36:35.1680102Z 
2022-05-18T04:36:35.1680180Z OK
2022-05-18T04:36:35.1680259Z 
2022-05-18T04:36:35.1680352Z Generating XML reports...
2022-05-18T04:36:35.1713612Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20220518043633.xml
2022-05-18T04:36:35.9267673Z Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper
2022-05-18T04:36:35.9276845Z 
2022-05-18T04:36:35.9277409Z Running tests...
2022-05-18T04:36:35.9277814Z ----------------------------------------------------------------------
2022-05-18T04:36:36.2129370Z   test_collectives_op_mismatch (__main__.ProcessGroupGlooWrapperTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 944
2022-05-18T04:36:36.2150573Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 945
2022-05-18T04:36:36.2173067Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 946
2022-05-18T04:36:36.2196245Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 947
2022-05-18T04:36:36.8426592Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:36:36.8508805Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:36:36.8598263Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:36:36.8688081Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:36:36.8806735Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:36:36.8998243Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:36:36.9099604Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T04:36:36.9100118Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:36:36.9100764Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:36:36.9101299Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:36:36.9101825Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:36:36.9111066Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:36:37.3230444Z ok (1.395s)
2022-05-18T04:36:37.3230678Z 
2022-05-18T04:36:37.3231196Z ----------------------------------------------------------------------
2022-05-18T04:36:37.3231921Z Ran 1 test in 1.395s
2022-05-18T04:36:37.3232037Z 
2022-05-18T04:36:37.3232099Z OK
2022-05-18T04:36:37.3232219Z 
2022-05-18T04:36:37.3232301Z Generating XML reports...
2022-05-18T04:36:37.3266268Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20220518043635.xml
2022-05-18T04:36:38.0861724Z Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper
2022-05-18T04:36:38.0872000Z 
2022-05-18T04:36:38.0872110Z Running tests...
2022-05-18T04:36:38.0872713Z ----------------------------------------------------------------------
2022-05-18T04:36:38.3684396Z   test_collectives_op_mismatch_cuda (__main__.ProcessGroupGlooWrapperTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1035
2022-05-18T04:36:38.3707070Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1036
2022-05-18T04:36:38.3729785Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1037
2022-05-18T04:36:38.3754070Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1038
2022-05-18T04:36:38.9716168Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:36:38.9933418Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:36:38.9997925Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:36:39.0079755Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:36:39.1783133Z skip: Need at least 4 CUDA devices (1.091s)
2022-05-18T04:36:39.1783430Z 
2022-05-18T04:36:39.1783966Z ----------------------------------------------------------------------
2022-05-18T04:36:39.1784267Z Ran 1 test in 1.091s
2022-05-18T04:36:39.1784385Z 
2022-05-18T04:36:39.1784460Z OK (skipped=1)
2022-05-18T04:36:39.1784571Z 
2022-05-18T04:36:39.1784645Z Generating XML reports...
2022-05-18T04:36:39.1817501Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20220518043638.xml
2022-05-18T04:36:39.9352953Z Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper
2022-05-18T04:36:39.9363426Z 
2022-05-18T04:36:39.9363744Z Running tests...
2022-05-18T04:36:39.9364394Z ----------------------------------------------------------------------
2022-05-18T04:36:40.2215214Z   test_collectives_op_mismatch_cuda_debug_mode (__main__.ProcessGroupGlooWrapperTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1090
2022-05-18T04:36:40.2236731Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1091
2022-05-18T04:36:40.2259106Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1092
2022-05-18T04:36:40.2282592Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1093
2022-05-18T04:36:40.8981249Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:36:40.8993733Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:36:40.9630299Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:36:40.9714076Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:36:41.1368194Z skip: Need at least 4 CUDA devices (1.200s)
2022-05-18T04:36:41.1368500Z 
2022-05-18T04:36:41.1368995Z ----------------------------------------------------------------------
2022-05-18T04:36:41.1369454Z Ran 1 test in 1.200s
2022-05-18T04:36:41.1369662Z 
2022-05-18T04:36:41.1369793Z OK (skipped=1)
2022-05-18T04:36:41.1369978Z 
2022-05-18T04:36:41.1370103Z Generating XML reports...
2022-05-18T04:36:41.1402361Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20220518043639.xml
2022-05-18T04:36:41.8963749Z Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper
2022-05-18T04:36:41.8973310Z 
2022-05-18T04:36:41.8973442Z Running tests...
2022-05-18T04:36:41.8974401Z ----------------------------------------------------------------------
2022-05-18T04:36:42.1790710Z   test_collectives_op_mismatch_debug_mode (__main__.ProcessGroupGlooWrapperTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1145
2022-05-18T04:36:42.1812440Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1146
2022-05-18T04:36:42.1834630Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1147
2022-05-18T04:36:42.1858069Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1148
2022-05-18T04:36:42.7983210Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-05-18T04:36:42.8109625Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-05-18T04:36:42.8156560Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-05-18T04:36:42.8207772Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-05-18T04:36:42.8824877Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:36:42.9025776Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-05-18T04:36:42.9027275Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-05-18T04:36:42.9028272Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:36:42.9028956Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-05-18T04:36:42.9029966Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:36:42.9030516Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:36:42.9031040Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-05-18T04:36:42.9641086Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-05-18T04:36:42.9741517Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-05-18T04:36:42.9742171Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-05-18T04:36:42.9743332Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
2022-05-18T04:36:42.9744022Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 3
2022-05-18T04:36:42.9744549Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
2022-05-18T04:36:42.9745077Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
2022-05-18T04:36:42.9745678Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
2022-05-18T04:36:43.2890687Z ok (1.391s)
2022-05-18T04:36:43.2890962Z 
2022-05-18T04:36:43.2891431Z ----------------------------------------------------------------------
2022-05-18T04:36:43.2891724Z Ran 1 test in 1.392s
2022-05-18T04:36:43.2891885Z 
2022-05-18T04:36:43.2891983Z OK
2022-05-18T04:36:43.2892095Z 
2022-05-18T04:36:43.2892212Z Generating XML reports...
2022-05-18T04:36:43.2925595Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20220518043641.xml
2022-05-18T04:36:44.0473092Z Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper
2022-05-18T04:36:44.0483066Z 
2022-05-18T04:36:44.0483324Z Running tests...
2022-05-18T04:36:44.0483938Z ----------------------------------------------------------------------
2022-05-18T04:36:44.0487724Z   test_collective_hang (__main__.ProcessGroupNCCLWrapperTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-05-18T04:36:44.0488091Z 
2022-05-18T04:36:44.0488411Z ----------------------------------------------------------------------
2022-05-18T04:36:44.0488713Z Ran 1 test in 0.000s
2022-05-18T04:36:44.0488828Z 
2022-05-18T04:36:44.0488931Z OK (skipped=1)
2022-05-18T04:36:44.0489072Z 
2022-05-18T04:36:44.0489157Z Generating XML reports...
2022-05-18T04:36:44.0514051Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupNCCLWrapperTest-20220518043644.xml
2022-05-18T04:36:44.7230771Z Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper
2022-05-18T04:36:44.7240324Z 
2022-05-18T04:36:44.7240452Z Running tests...
2022-05-18T04:36:44.7241024Z ----------------------------------------------------------------------
2022-05-18T04:36:44.7245501Z   test_collective_shape_mismatch (__main__.ProcessGroupNCCLWrapperTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-05-18T04:36:44.7246089Z 
2022-05-18T04:36:44.7246368Z ----------------------------------------------------------------------
2022-05-18T04:36:44.7246614Z Ran 1 test in 0.000s
2022-05-18T04:36:44.7246728Z 
2022-05-18T04:36:44.7246801Z OK (skipped=1)
2022-05-18T04:36:44.7246909Z 
2022-05-18T04:36:44.7246981Z Generating XML reports...
2022-05-18T04:36:44.7276220Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupNCCLWrapperTest-20220518043644.xml
2022-05-18T04:36:45.3869790Z Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper
2022-05-18T04:36:45.3880265Z 
2022-05-18T04:36:45.3880704Z Running tests...
2022-05-18T04:36:45.3881121Z ----------------------------------------------------------------------
2022-05-18T04:36:45.3886020Z   test_collective_shape_mismatch_debug_mode (__main__.ProcessGroupNCCLWrapperTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-05-18T04:36:45.3886552Z 
2022-05-18T04:36:45.3886872Z ----------------------------------------------------------------------
2022-05-18T04:36:45.3887121Z Ran 1 test in 0.001s
2022-05-18T04:36:45.3887236Z 
2022-05-18T04:36:45.3887309Z OK (skipped=1)
2022-05-18T04:36:45.3887418Z 
2022-05-18T04:36:45.3887489Z Generating XML reports...
2022-05-18T04:36:45.3912435Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupNCCLWrapperTest-20220518043645.xml
2022-05-18T04:36:46.0457526Z Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper
2022-05-18T04:36:46.0466686Z 
2022-05-18T04:36:46.0467223Z Running tests...
2022-05-18T04:36:46.0467640Z ----------------------------------------------------------------------
2022-05-18T04:36:46.0471994Z   test_collectives_op_mismatch (__main__.ProcessGroupNCCLWrapperTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-05-18T04:36:46.0472448Z 
2022-05-18T04:36:46.0472824Z ----------------------------------------------------------------------
2022-05-18T04:36:46.0473236Z Ran 1 test in 0.001s
2022-05-18T04:36:46.0473452Z 
2022-05-18T04:36:46.0473590Z OK (skipped=1)
2022-05-18T04:36:46.0473781Z 
2022-05-18T04:36:46.0473876Z Generating XML reports...
2022-05-18T04:36:46.0497108Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupNCCLWrapperTest-20220518043646.xml
2022-05-18T04:36:46.7085336Z Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper
2022-05-18T04:36:46.7094585Z 
2022-05-18T04:36:46.7094694Z Running tests...
2022-05-18T04:36:46.7095717Z ----------------------------------------------------------------------
2022-05-18T04:36:46.7099803Z   test_collectives_op_mismatch_debug_mode (__main__.ProcessGroupNCCLWrapperTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-05-18T04:36:46.7100224Z 
2022-05-18T04:36:46.7100725Z ----------------------------------------------------------------------
2022-05-18T04:36:46.7101074Z Ran 1 test in 0.000s
2022-05-18T04:36:46.7101293Z 
2022-05-18T04:36:46.7101436Z OK (skipped=1)
2022-05-18T04:36:46.7101643Z 
2022-05-18T04:36:46.7101763Z Generating XML reports...
2022-05-18T04:36:46.7135134Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupNCCLWrapperTest-20220518043646.xml
2022-05-18T04:36:46.9286094Z Running distributed/test_store ... [2022-05-18 04:36:46.928229]
2022-05-18T04:36:46.9286668Z Executing ['/opt/conda/bin/python', 'distributed/test_store.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-05-18 04:36:46.928315]
2022-05-18T04:36:47.4858873Z test_compare_set (__main__.FileStoreTest)
2022-05-18T04:36:47.4859545Z test_set_get (__main__.FileStoreTest)
2022-05-18T04:36:47.4860080Z test_compare_set (__main__.HashStoreTest)
2022-05-18T04:36:47.4860440Z test_set_get (__main__.HashStoreTest)
2022-05-18T04:36:47.4860966Z test_compare_set (__main__.PrefixFileStoreTest)
2022-05-18T04:36:47.4861435Z test_set_get (__main__.PrefixFileStoreTest)
2022-05-18T04:36:47.4861802Z test_compare_set (__main__.PrefixTCPStoreTest)
2022-05-18T04:36:47.4862096Z test_set_get (__main__.PrefixTCPStoreTest)
2022-05-18T04:36:47.4862345Z test_set_get (__main__.PythonStoreTest)
2022-05-18T04:36:47.4862634Z test_nominal (__main__.RendezvousEnvTest)
2022-05-18T04:36:47.4863104Z test_common_errors (__main__.RendezvousFileTest)
2022-05-18T04:36:47.4863363Z test_nominal (__main__.RendezvousFileTest)
2022-05-18T04:36:47.4863592Z test_common_errors (__main__.RendezvousTCPTest)
2022-05-18T04:36:47.4863888Z test_dns_timeout (__main__.RendezvousTCPTest)
2022-05-18T04:36:47.4864135Z test_nominal (__main__.RendezvousTCPTest)
2022-05-18T04:36:47.4864377Z test_tcp_store_timeout_set (__main__.RendezvousTCPTest)
2022-05-18T04:36:47.4864688Z test_unknown_handler (__main__.RendezvousTest)
2022-05-18T04:36:47.4864941Z test_address_already_in_use (__main__.TCPStoreTest)
2022-05-18T04:36:47.4865186Z test_compare_set (__main__.TCPStoreTest)
2022-05-18T04:36:47.4865480Z test_init_pg_and_rpc_with_same_socket (__main__.TCPStoreTest)
2022-05-18T04:36:47.4865758Z test_multi_worker_with_fixed_world_size (__main__.TCPStoreTest)
2022-05-18T04:36:47.4866038Z test_multi_worker_with_nonfixed_world_size (__main__.TCPStoreTest)
2022-05-18T04:36:47.4866341Z test_multitenancy (__main__.TCPStoreTest)
2022-05-18T04:36:47.4866579Z test_numkeys_delkeys (__main__.TCPStoreTest)
2022-05-18T04:36:47.4866806Z test_set_get (__main__.TCPStoreTest)
2022-05-18T04:36:48.0399292Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-05-18T04:36:48.0408683Z 
2022-05-18T04:36:48.0408797Z Running tests...
2022-05-18T04:36:48.0409347Z ----------------------------------------------------------------------
2022-05-18T04:36:48.3140960Z   test_compare_set (__main__.FileStoreTest) ... ok (0.273s)
2022-05-18T04:36:48.3141345Z 
2022-05-18T04:36:48.3141819Z ----------------------------------------------------------------------
2022-05-18T04:36:48.3142098Z Ran 1 test in 0.273s
2022-05-18T04:36:48.3142212Z 
2022-05-18T04:36:48.3142273Z OK
2022-05-18T04:36:48.3142364Z 
2022-05-18T04:36:48.3142437Z Generating XML reports...
2022-05-18T04:36:48.3164965Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-FileStoreTest-20220518043648.xml
2022-05-18T04:36:49.0232608Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-05-18T04:36:49.0241785Z 
2022-05-18T04:36:49.0241917Z Running tests...
2022-05-18T04:36:49.0242307Z ----------------------------------------------------------------------
2022-05-18T04:36:49.3006845Z   test_set_get (__main__.FileStoreTest) ... ok (0.276s)
2022-05-18T04:36:49.3007163Z 
2022-05-18T04:36:49.3007833Z ----------------------------------------------------------------------
2022-05-18T04:36:49.3008263Z Ran 1 test in 0.276s
2022-05-18T04:36:49.3008467Z 
2022-05-18T04:36:49.3008574Z OK
2022-05-18T04:36:49.3008834Z 
2022-05-18T04:36:49.3008997Z Generating XML reports...
2022-05-18T04:36:49.3032606Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-FileStoreTest-20220518043649.xml
2022-05-18T04:36:50.0108804Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-05-18T04:36:50.0119029Z 
2022-05-18T04:36:50.0119341Z Running tests...
2022-05-18T04:36:50.0119963Z ----------------------------------------------------------------------
2022-05-18T04:36:50.2858278Z   test_compare_set (__main__.HashStoreTest) ... ok (0.274s)
2022-05-18T04:36:50.2858582Z 
2022-05-18T04:36:50.2859056Z ----------------------------------------------------------------------
2022-05-18T04:36:50.2859517Z Ran 1 test in 0.274s
2022-05-18T04:36:50.2859710Z 
2022-05-18T04:36:50.2859828Z OK
2022-05-18T04:36:50.2859993Z 
2022-05-18T04:36:50.2860146Z Generating XML reports...
2022-05-18T04:36:50.2884004Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-HashStoreTest-20220518043650.xml
2022-05-18T04:36:50.9952408Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-05-18T04:36:50.9962186Z 
2022-05-18T04:36:50.9962650Z Running tests...
2022-05-18T04:36:50.9963063Z ----------------------------------------------------------------------
2022-05-18T04:36:51.2700121Z   test_set_get (__main__.HashStoreTest) ... ok (0.274s)
2022-05-18T04:36:51.2700437Z 
2022-05-18T04:36:51.2700901Z ----------------------------------------------------------------------
2022-05-18T04:36:51.2701365Z Ran 1 test in 0.274s
2022-05-18T04:36:51.2701571Z 
2022-05-18T04:36:51.2701678Z OK
2022-05-18T04:36:51.2701829Z 
2022-05-18T04:36:51.2701988Z Generating XML reports...
2022-05-18T04:36:51.2731964Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-HashStoreTest-20220518043650.xml
2022-05-18T04:36:51.9832983Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-05-18T04:36:51.9842952Z 
2022-05-18T04:36:51.9843379Z Running tests...
2022-05-18T04:36:51.9843834Z ----------------------------------------------------------------------
2022-05-18T04:36:52.2613820Z   test_compare_set (__main__.PrefixFileStoreTest) ... ok (0.277s)
2022-05-18T04:36:52.2614184Z 
2022-05-18T04:36:52.2614658Z ----------------------------------------------------------------------
2022-05-18T04:36:52.2615126Z Ran 1 test in 0.277s
2022-05-18T04:36:52.2615328Z 
2022-05-18T04:36:52.2615440Z OK
2022-05-18T04:36:52.2615605Z 
2022-05-18T04:36:52.2615743Z Generating XML reports...
2022-05-18T04:36:52.2640176Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-PrefixFileStoreTest-20220518043651.xml
2022-05-18T04:36:52.9896795Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-05-18T04:36:52.9906467Z 
2022-05-18T04:36:52.9906575Z Running tests...
2022-05-18T04:36:52.9907645Z ----------------------------------------------------------------------
2022-05-18T04:36:53.2673512Z   test_set_get (__main__.PrefixFileStoreTest) ... ok (0.276s)
2022-05-18T04:36:53.2673858Z 
2022-05-18T04:36:53.2674279Z ----------------------------------------------------------------------
2022-05-18T04:36:53.2674528Z Ran 1 test in 0.277s
2022-05-18T04:36:53.2674644Z 
2022-05-18T04:36:53.2674692Z OK
2022-05-18T04:36:53.2674783Z 
2022-05-18T04:36:53.2674868Z Generating XML reports...
2022-05-18T04:36:53.2697632Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-PrefixFileStoreTest-20220518043652.xml
2022-05-18T04:36:53.9792735Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-05-18T04:36:53.9801554Z 
2022-05-18T04:36:53.9801696Z Running tests...
2022-05-18T04:36:53.9802221Z ----------------------------------------------------------------------
2022-05-18T04:36:54.2546294Z   test_compare_set (__main__.PrefixTCPStoreTest) ... ok (0.274s)
2022-05-18T04:36:54.2546590Z 
2022-05-18T04:36:54.2547085Z ----------------------------------------------------------------------
2022-05-18T04:36:54.2547364Z Ran 1 test in 0.274s
2022-05-18T04:36:54.2547465Z 
2022-05-18T04:36:54.2547527Z OK
2022-05-18T04:36:54.2547619Z 
2022-05-18T04:36:54.2547707Z Generating XML reports...
2022-05-18T04:36:54.2569409Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-PrefixTCPStoreTest-20220518043653.xml
2022-05-18T04:36:54.9674651Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-05-18T04:36:54.9683327Z 
2022-05-18T04:36:54.9683460Z Running tests...
2022-05-18T04:36:54.9683842Z ----------------------------------------------------------------------
2022-05-18T04:36:55.2426406Z   test_set_get (__main__.PrefixTCPStoreTest) ... ok (0.274s)
2022-05-18T04:36:55.2426745Z 
2022-05-18T04:36:55.2427221Z ----------------------------------------------------------------------
2022-05-18T04:36:55.2427501Z Ran 1 test in 0.274s
2022-05-18T04:36:55.2427619Z 
2022-05-18T04:36:55.2427667Z OK
2022-05-18T04:36:55.2427761Z 
2022-05-18T04:36:55.2427853Z Generating XML reports...
2022-05-18T04:36:55.2451685Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-PrefixTCPStoreTest-20220518043654.xml
2022-05-18T04:36:55.9647611Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-05-18T04:36:55.9656766Z 
2022-05-18T04:36:55.9656882Z Running tests...
2022-05-18T04:36:55.9657478Z ----------------------------------------------------------------------
2022-05-18T04:36:56.2375754Z   test_set_get (__main__.PythonStoreTest) ... ok (0.272s)
2022-05-18T04:36:56.2376031Z 
2022-05-18T04:36:56.2376345Z ----------------------------------------------------------------------
2022-05-18T04:36:56.2376595Z Ran 1 test in 0.272s
2022-05-18T04:36:56.2376711Z 
2022-05-18T04:36:56.2376789Z OK
2022-05-18T04:36:56.2376869Z 
2022-05-18T04:36:56.2376963Z Generating XML reports...
2022-05-18T04:36:56.2399717Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-PythonStoreTest-20220518043655.xml
2022-05-18T04:36:56.9459010Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-05-18T04:36:56.9468169Z 
2022-05-18T04:36:56.9468654Z Running tests...
2022-05-18T04:36:56.9469049Z ----------------------------------------------------------------------
2022-05-18T04:36:57.2189537Z   test_nominal (__main__.RendezvousEnvTest) ... ok (0.272s)
2022-05-18T04:36:57.2189866Z 
2022-05-18T04:36:57.2190185Z ----------------------------------------------------------------------
2022-05-18T04:36:57.2190452Z Ran 1 test in 0.272s
2022-05-18T04:36:57.2190554Z 
2022-05-18T04:36:57.2190617Z OK
2022-05-18T04:36:57.2190709Z 
2022-05-18T04:36:57.2190799Z Generating XML reports...
2022-05-18T04:36:57.2212956Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-RendezvousEnvTest-20220518043656.xml
2022-05-18T04:36:57.9405972Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-05-18T04:36:57.9415946Z 
2022-05-18T04:36:57.9416360Z Running tests...
2022-05-18T04:36:57.9416775Z ----------------------------------------------------------------------
2022-05-18T04:36:58.2144697Z   test_common_errors (__main__.RendezvousFileTest) ... ok (0.273s)
2022-05-18T04:36:58.2145021Z 
2022-05-18T04:36:58.2145493Z ----------------------------------------------------------------------
2022-05-18T04:36:58.2145936Z Ran 1 test in 0.273s
2022-05-18T04:36:58.2146148Z 
2022-05-18T04:36:58.2146263Z OK
2022-05-18T04:36:58.2146425Z 
2022-05-18T04:36:58.2146571Z Generating XML reports...
2022-05-18T04:36:58.2169831Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-RendezvousFileTest-20220518043657.xml
2022-05-18T04:36:58.9331885Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-05-18T04:36:58.9340893Z 
2022-05-18T04:36:58.9341039Z Running tests...
2022-05-18T04:36:58.9341410Z ----------------------------------------------------------------------
2022-05-18T04:36:59.2097586Z   test_nominal (__main__.RendezvousFileTest) ... ok (0.275s)
2022-05-18T04:36:59.2097885Z 
2022-05-18T04:36:59.2098366Z ----------------------------------------------------------------------
2022-05-18T04:36:59.2098690Z Ran 1 test in 0.276s
2022-05-18T04:36:59.2098792Z 
2022-05-18T04:36:59.2098855Z OK
2022-05-18T04:36:59.2098948Z 
2022-05-18T04:36:59.2099046Z Generating XML reports...
2022-05-18T04:36:59.2121089Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-RendezvousFileTest-20220518043658.xml
2022-05-18T04:36:59.9219784Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-05-18T04:36:59.9229164Z 
2022-05-18T04:36:59.9229645Z Running tests...
2022-05-18T04:36:59.9230301Z ----------------------------------------------------------------------
2022-05-18T04:37:00.1967604Z   test_common_errors (__main__.RendezvousTCPTest) ... ok (0.274s)
2022-05-18T04:37:00.1967903Z 
2022-05-18T04:37:00.1968257Z ----------------------------------------------------------------------
2022-05-18T04:37:00.1968508Z Ran 1 test in 0.274s
2022-05-18T04:37:00.1968622Z 
2022-05-18T04:37:00.1968682Z OK
2022-05-18T04:37:00.1968774Z 
2022-05-18T04:37:00.1968859Z Generating XML reports...
2022-05-18T04:37:00.1992215Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-RendezvousTCPTest-20220518043659.xml
2022-05-18T04:37:00.9058493Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-05-18T04:37:00.9068443Z 
2022-05-18T04:37:00.9068858Z Running tests...
2022-05-18T04:37:00.9069362Z ----------------------------------------------------------------------
2022-05-18T04:37:01.2249135Z   test_dns_timeout (__main__.RendezvousTCPTest) ... [W socket.cpp:558] [c10d] The IPv6 network addresses of (dnsnotexist, 23456) cannot be retrieved (gai error: -2 - Name or service not known).
2022-05-18T04:37:01.2249872Z [E socket.cpp:793] [c10d] The client socket has timed out after 1s while trying to connect to (dnsnotexist, 23456).
2022-05-18T04:37:01.2250933Z ok (0.318s)
2022-05-18T04:37:01.2251867Z 
2022-05-18T04:37:01.2252237Z ----------------------------------------------------------------------
2022-05-18T04:37:01.2252666Z Ran 1 test in 0.318s
2022-05-18T04:37:01.2252851Z 
2022-05-18T04:37:01.2252948Z OK
2022-05-18T04:37:01.2253094Z 
2022-05-18T04:37:01.2253235Z Generating XML reports...
2022-05-18T04:37:01.2278653Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-RendezvousTCPTest-20220518043700.xml
2022-05-18T04:37:01.9454555Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-05-18T04:37:01.9464667Z 
2022-05-18T04:37:01.9465102Z Running tests...
2022-05-18T04:37:01.9465767Z ----------------------------------------------------------------------
2022-05-18T04:37:02.2216369Z   test_nominal (__main__.RendezvousTCPTest) ... ok (0.275s)
2022-05-18T04:37:02.2216723Z 
2022-05-18T04:37:02.2217202Z ----------------------------------------------------------------------
2022-05-18T04:37:02.2217684Z Ran 1 test in 0.275s
2022-05-18T04:37:02.2217892Z 
2022-05-18T04:37:02.2218001Z OK
2022-05-18T04:37:02.2218153Z 
2022-05-18T04:37:02.2218306Z Generating XML reports...
2022-05-18T04:37:02.2242213Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-RendezvousTCPTest-20220518043701.xml
2022-05-18T04:37:02.9470602Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-05-18T04:37:02.9480101Z 
2022-05-18T04:37:02.9480181Z Running tests...
2022-05-18T04:37:02.9480936Z ----------------------------------------------------------------------
2022-05-18T04:37:13.4698830Z   test_tcp_store_timeout_set (__main__.RendezvousTCPTest) ... ok (10.522s)
2022-05-18T04:37:13.4699211Z 
2022-05-18T04:37:13.4699628Z ----------------------------------------------------------------------
2022-05-18T04:37:13.4700137Z Ran 1 test in 10.522s
2022-05-18T04:37:13.4700253Z 
2022-05-18T04:37:13.4700300Z OK
2022-05-18T04:37:13.4700393Z 
2022-05-18T04:37:13.4700543Z Generating XML reports...
2022-05-18T04:37:13.4726475Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-RendezvousTCPTest-20220518043702.xml
2022-05-18T04:37:14.2114180Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-05-18T04:37:14.2122682Z 
2022-05-18T04:37:14.2122819Z Running tests...
2022-05-18T04:37:14.2123373Z ----------------------------------------------------------------------
2022-05-18T04:37:14.4834347Z   test_unknown_handler (__main__.RendezvousTest) ... ok (0.271s)
2022-05-18T04:37:14.4834682Z 
2022-05-18T04:37:14.4835145Z ----------------------------------------------------------------------
2022-05-18T04:37:14.4835406Z Ran 1 test in 0.271s
2022-05-18T04:37:14.4835520Z 
2022-05-18T04:37:14.4835580Z OK
2022-05-18T04:37:14.4835680Z 
2022-05-18T04:37:14.4835756Z Generating XML reports...
2022-05-18T04:37:14.4858427Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-RendezvousTest-20220518043714.xml
2022-05-18T04:37:15.1930005Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-05-18T04:37:15.1939155Z 
2022-05-18T04:37:15.1939274Z Running tests...
2022-05-18T04:37:15.1939884Z ----------------------------------------------------------------------
2022-05-18T04:37:15.4671390Z   test_address_already_in_use (__main__.TCPStoreTest) ... [W socket.cpp:401] [c10d] The server socket has failed to bind to [::]:44855 (errno: 98 - Address already in use).
2022-05-18T04:37:15.4680272Z [W socket.cpp:401] [c10d] The server socket has failed to bind to 0.0.0.0:44855 (errno: 98 - Address already in use).
2022-05-18T04:37:15.4680864Z [E socket.cpp:435] [c10d] The server socket has failed to listen on any local network address.
2022-05-18T04:37:15.4683921Z ok (0.274s)
2022-05-18T04:37:15.4684297Z 
2022-05-18T04:37:15.4685113Z ----------------------------------------------------------------------
2022-05-18T04:37:15.4685490Z Ran 1 test in 0.275s
2022-05-18T04:37:15.4685610Z 
2022-05-18T04:37:15.4685671Z OK
2022-05-18T04:37:15.4685773Z 
2022-05-18T04:37:15.4685864Z Generating XML reports...
2022-05-18T04:37:15.4710101Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20220518043715.xml
2022-05-18T04:37:16.1864346Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-05-18T04:37:16.1873746Z 
2022-05-18T04:37:16.1873830Z Running tests...
2022-05-18T04:37:16.1874273Z ----------------------------------------------------------------------
2022-05-18T04:37:16.4643446Z   test_compare_set (__main__.TCPStoreTest) ... ok (0.277s)
2022-05-18T04:37:16.4643784Z 
2022-05-18T04:37:16.4644145Z ----------------------------------------------------------------------
2022-05-18T04:37:16.4644398Z Ran 1 test in 0.277s
2022-05-18T04:37:16.4644526Z 
2022-05-18T04:37:16.4644589Z OK
2022-05-18T04:37:16.4644683Z 
2022-05-18T04:37:16.4644768Z Generating XML reports...
2022-05-18T04:37:16.4667859Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20220518043716.xml
2022-05-18T04:37:17.1743215Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-05-18T04:37:17.1752397Z 
2022-05-18T04:37:17.1752481Z Running tests...
2022-05-18T04:37:17.1753470Z ----------------------------------------------------------------------
2022-05-18T04:37:17.4541137Z   test_init_pg_and_rpc_with_same_socket (__main__.TCPStoreTest) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-05-18T04:37:17.4541826Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-05-18T04:37:17.5248516Z ok (0.349s)
2022-05-18T04:37:17.5248684Z 
2022-05-18T04:37:17.5249007Z ----------------------------------------------------------------------
2022-05-18T04:37:17.5249519Z Ran 1 test in 0.350s
2022-05-18T04:37:17.5249637Z 
2022-05-18T04:37:17.5249698Z OK
2022-05-18T04:37:17.5249791Z 
2022-05-18T04:37:17.5249941Z Generating XML reports...
2022-05-18T04:37:17.5274804Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20220518043717.xml
2022-05-18T04:37:18.2357193Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-05-18T04:37:18.2366343Z 
2022-05-18T04:37:18.2366495Z Running tests...
2022-05-18T04:37:18.2366881Z ----------------------------------------------------------------------
2022-05-18T04:37:18.5139370Z   test_multi_worker_with_fixed_world_size (__main__.TCPStoreTest) ... ok (0.277s)
2022-05-18T04:37:18.5139581Z 
2022-05-18T04:37:18.5139880Z ----------------------------------------------------------------------
2022-05-18T04:37:18.5140140Z Ran 1 test in 0.277s
2022-05-18T04:37:18.5140256Z 
2022-05-18T04:37:18.5140333Z OK
2022-05-18T04:37:18.5140426Z 
2022-05-18T04:37:18.5140502Z Generating XML reports...
2022-05-18T04:37:18.5163508Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20220518043718.xml
2022-05-18T04:37:19.2259772Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-05-18T04:37:19.2268681Z 
2022-05-18T04:37:19.2268784Z Running tests...
2022-05-18T04:37:19.2269811Z ----------------------------------------------------------------------
2022-05-18T04:37:19.5014800Z   test_multi_worker_with_nonfixed_world_size (__main__.TCPStoreTest) ... ok (0.274s)
2022-05-18T04:37:19.5015177Z 
2022-05-18T04:37:19.5015604Z ----------------------------------------------------------------------
2022-05-18T04:37:19.5015835Z Ran 1 test in 0.275s
2022-05-18T04:37:19.5015949Z 
2022-05-18T04:37:19.5016013Z OK
2022-05-18T04:37:19.5016106Z 
2022-05-18T04:37:19.5016192Z Generating XML reports...
2022-05-18T04:37:19.5040711Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20220518043719.xml
2022-05-18T04:37:20.2142412Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-05-18T04:37:20.2151771Z 
2022-05-18T04:37:20.2151944Z Running tests...
2022-05-18T04:37:20.2152316Z ----------------------------------------------------------------------
2022-05-18T04:37:20.4880370Z   test_multitenancy (__main__.TCPStoreTest) ... ok (0.273s)
2022-05-18T04:37:20.4880744Z 
2022-05-18T04:37:20.4881198Z ----------------------------------------------------------------------
2022-05-18T04:37:20.4881671Z Ran 1 test in 0.273s
2022-05-18T04:37:20.4881876Z 
2022-05-18T04:37:20.4881984Z OK
2022-05-18T04:37:20.4882136Z 
2022-05-18T04:37:20.4882293Z Generating XML reports...
2022-05-18T04:37:20.4905237Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20220518043720.xml
2022-05-18T04:37:21.1983450Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-05-18T04:37:21.1992336Z 
2022-05-18T04:37:21.1992462Z Running tests...
2022-05-18T04:37:21.1993149Z ----------------------------------------------------------------------
2022-05-18T04:37:23.4865554Z   test_numkeys_delkeys (__main__.TCPStoreTest) ... ok (2.287s)
2022-05-18T04:37:23.4865928Z 
2022-05-18T04:37:23.4866409Z ----------------------------------------------------------------------
2022-05-18T04:37:23.4866661Z Ran 1 test in 2.287s
2022-05-18T04:37:23.4866774Z 
2022-05-18T04:37:23.4866822Z OK
2022-05-18T04:37:23.4866914Z 
2022-05-18T04:37:23.4867001Z Generating XML reports...
2022-05-18T04:37:23.4892150Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20220518043721.xml
2022-05-18T04:37:24.2194840Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-05-18T04:37:24.2203807Z 
2022-05-18T04:37:24.2203943Z Running tests...
2022-05-18T04:37:24.2204646Z ----------------------------------------------------------------------
2022-05-18T04:37:24.4955243Z   test_set_get (__main__.TCPStoreTest) ... ok (0.275s)
2022-05-18T04:37:24.4955529Z 
2022-05-18T04:37:24.4956237Z ----------------------------------------------------------------------
2022-05-18T04:37:24.4956713Z Ran 1 test in 0.275s
2022-05-18T04:37:24.4956920Z 
2022-05-18T04:37:24.4957028Z OK
2022-05-18T04:37:24.4957193Z 
2022-05-18T04:37:24.4957348Z Generating XML reports...
2022-05-18T04:37:24.4980555Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20220518043724.xml
2022-05-18T04:37:24.9779175Z 
2022-05-18T04:37:24.9779529Z real	64m24.149s
2022-05-18T04:37:24.9779840Z user	124m45.952s
2022-05-18T04:37:24.9780161Z sys	43m36.293s
2022-05-18T04:37:24.9780493Z + assert_git_not_dirty
2022-05-18T04:37:24.9780923Z + [[ linux-xenial-py3.7-gcc5.4-distributed != *rocm* ]]
2022-05-18T04:37:24.9781273Z + [[ linux-xenial-py3.7-gcc5.4-distributed != *xla* ]]
2022-05-18T04:37:24.9783414Z ++ git status --porcelain
2022-05-18T04:37:25.4982523Z + git_status=
2022-05-18T04:37:25.4983086Z + [[ -n '' ]]
2022-05-18T04:37:25.4983528Z + [[ linux-xenial-py3.7-gcc5.4-distributed == *cuda* ]]
2022-05-18T04:37:25.4983762Z + [[ 1 == 1 ]]
2022-05-18T04:37:25.4983977Z + test_rpc
2022-05-18T04:37:25.4984268Z + [[ linux-xenial-py3.7-gcc5.4-distributed != *rocm* ]]
2022-05-18T04:37:25.4984551Z + echo 'Testing RPC C++ tests'
2022-05-18T04:37:25.4984736Z Testing RPC C++ tests
2022-05-18T04:37:25.4985557Z + ln -sf /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch.so /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_global_deps.so /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_python.so /opt/conda/lib/python3.7/site-packages/torch/lib/libtorchbind_test.so /opt/conda/lib/python3.7/site-packages/torch/bin
2022-05-18T04:37:25.4994537Z + ln -sf /opt/conda/lib/python3.7/site-packages/torch/lib/libc10.so /opt/conda/lib/python3.7/site-packages/torch/bin
2022-05-18T04:37:25.5002686Z + ln -sf '/opt/conda/lib/python3.7/site-packages/torch/lib/libtbb*' /opt/conda/lib/python3.7/site-packages/torch/bin
2022-05-18T04:37:25.5009973Z + TEST_REPORTS_DIR=test/test-reports/cpp-rpc/test_rpc
2022-05-18T04:37:25.5010303Z + mkdir -p test/test-reports/cpp-rpc/test_rpc
2022-05-18T04:37:25.5021040Z + /opt/conda/lib/python3.7/site-packages/torch/bin/test_cpp_rpc --gtest_output=xml:test/test-reports/cpp-rpc/test_rpc/test_cpp_rpc.xml
2022-05-18T04:37:25.6630039Z CUDA not available. Disabling CUDA and MultiCUDA tests
2022-05-18T04:37:25.6630662Z Note: Google Test filter = *-*_CUDA:*_MultiCUDA
2022-05-18T04:37:25.6630980Z [==========] Running 8 tests from 3 test suites.
2022-05-18T04:37:25.6631276Z [----------] Global test environment set-up.
2022-05-18T04:37:25.6631630Z [----------] 4 tests from WireSerialize
2022-05-18T04:37:25.6631900Z [ RUN      ] WireSerialize.Base
2022-05-18T04:37:25.6831084Z [       OK ] WireSerialize.Base (20 ms)
2022-05-18T04:37:25.6831438Z [ RUN      ] WireSerialize.RecopySparseTensors
2022-05-18T04:37:25.6905724Z [       OK ] WireSerialize.RecopySparseTensors (7 ms)
2022-05-18T04:37:25.6906073Z [ RUN      ] WireSerialize.CloneSparseTensors
2022-05-18T04:37:25.6973736Z [       OK ] WireSerialize.CloneSparseTensors (6 ms)
2022-05-18T04:37:25.6974046Z [ RUN      ] WireSerialize.Errors
2022-05-18T04:37:25.6996429Z [       OK ] WireSerialize.Errors (2 ms)
2022-05-18T04:37:25.6996801Z [----------] 4 tests from WireSerialize (36 ms total)
2022-05-18T04:37:25.6996963Z 
2022-05-18T04:37:25.6997124Z [----------] 1 test from TestE2ETensorPipe
2022-05-18T04:37:25.6997445Z [ RUN      ] TestE2ETensorPipe.TestTrainingLoop
2022-05-18T04:37:26.1141436Z [W tensorpipe_agent.cpp:728] RPC agent for worker encountered error when reading incoming request from worker: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2022-05-18T04:37:26.1150147Z [       OK ] TestE2ETensorPipe.TestTrainingLoop (415 ms)
2022-05-18T04:37:26.1150699Z [----------] 1 test from TestE2ETensorPipe (415 ms total)
2022-05-18T04:37:26.1150880Z 
2022-05-18T04:37:26.1151181Z [----------] 3 tests from TensorpipeSerialize
2022-05-18T04:37:26.1151488Z [ RUN      ] TensorpipeSerialize.Base
2022-05-18T04:37:26.1151792Z [       OK ] TensorpipeSerialize.Base (0 ms)
2022-05-18T04:37:26.1152123Z [ RUN      ] TensorpipeSerialize.RecopySparseTensors
2022-05-18T04:37:26.1214021Z [       OK ] TensorpipeSerialize.RecopySparseTensors (6 ms)
2022-05-18T04:37:26.1214555Z [ RUN      ] TensorpipeSerialize.NoDeleterTensors
2022-05-18T04:37:26.1215099Z [       OK ] TensorpipeSerialize.NoDeleterTensors (0 ms)
2022-05-18T04:37:26.1215581Z [----------] 3 tests from TensorpipeSerialize (6 ms total)
2022-05-18T04:37:26.1215811Z 
2022-05-18T04:37:26.1216041Z [----------] Global test environment tear-down
2022-05-18T04:37:26.1217917Z [==========] 8 tests from 3 test suites ran. (458 ms total)
2022-05-18T04:37:26.1218293Z [  PASSED  ] 8 tests.
2022-05-18T04:37:26.1218458Z 
2022-05-18T04:37:26.1218630Z   YOU HAVE 1 DISABLED TEST
2022-05-18T04:37:26.1218792Z 
2022-05-18T04:37:26.1629711Z + cleanup
2022-05-18T04:37:26.1630106Z + retcode=0
2022-05-18T04:37:26.1630299Z + set +x
2022-05-18T04:37:26.1630483Z EXITED_USER_LAND
2022-05-18T04:37:26.1693297Z ##[group]Run pytorch/pytorch/.github/actions/get-workflow-job-id@master
2022-05-18T04:37:26.1693548Z with:
2022-05-18T04:37:26.1693974Z   github-token: ***
2022-05-18T04:37:26.1694132Z env:
2022-05-18T04:37:26.1694289Z   IN_CI: 1
2022-05-18T04:37:26.1694450Z   IS_GHA: 1
2022-05-18T04:37:26.1694617Z   GIT_DEFAULT_BRANCH: master
2022-05-18T04:37:26.1694802Z ##[endgroup]
2022-05-18T04:37:26.1718429Z ##[group]Run nick-fields/retry@71062288b76e2b6214ebde0e673ce0de1755740a
2022-05-18T04:37:26.1718659Z with:
2022-05-18T04:37:26.1718831Z   shell: bash
2022-05-18T04:37:26.1718993Z   timeout_minutes: 10
2022-05-18T04:37:26.1719173Z   max_attempts: 5
2022-05-18T04:37:26.1719352Z   retry_wait_seconds: 30
2022-05-18T04:37:26.1719716Z   command: set -x
python3 -m pip install requests==2.26.0
GHA_WORKFLOW_JOB_ID=$(python3 .github/scripts/get_workflow_job_id.py "${GITHUB_RUN_ID}" "${RUNNER_NAME}")
echo "::set-output name=job-id::${GHA_WORKFLOW_JOB_ID}"

2022-05-18T04:37:26.1720135Z   polling_interval_seconds: 1
2022-05-18T04:37:26.1720331Z   warning_on_retry: true
2022-05-18T04:37:26.1720508Z   continue_on_error: false
2022-05-18T04:37:26.1720678Z env:
2022-05-18T04:37:26.1720829Z   IN_CI: 1
2022-05-18T04:37:26.1720978Z   IS_GHA: 1
2022-05-18T04:37:26.1721156Z   GIT_DEFAULT_BRANCH: master
2022-05-18T04:37:26.1721462Z   GITHUB_TOKEN: ***
2022-05-18T04:37:26.1721623Z ##[endgroup]
2022-05-18T04:37:26.2026620Z 
2022-05-18T04:37:26.2079202Z + python3 -m pip install requests==2.26.0
2022-05-18T04:37:26.4104172Z Defaulting to user installation because normal site-packages is not writeable
2022-05-18T04:37:26.4263584Z Requirement already satisfied: requests==2.26.0 in /home/ec2-user/.local/lib/python3.7/site-packages (2.26.0)
2022-05-18T04:37:26.4390182Z Requirement already satisfied: urllib3<1.27,>=1.21.1 in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.26.0) (1.26.9)
2022-05-18T04:37:26.4533992Z Requirement already satisfied: idna<4,>=2.5; python_version >= "3" in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.26.0) (3.3)
2022-05-18T04:37:26.4544924Z Requirement already satisfied: charset-normalizer~=2.0.0; python_version >= "3" in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.26.0) (2.0.12)
2022-05-18T04:37:26.4562141Z Requirement already satisfied: certifi>=2017.4.17 in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.26.0) (2021.10.8)
2022-05-18T04:37:26.5225090Z ++ python3 .github/scripts/get_workflow_job_id.py 2342799944 i-0244965c0907218b7
2022-05-18T04:37:28.6505083Z + GHA_WORKFLOW_JOB_ID=6482431846
2022-05-18T04:37:28.6505629Z + echo '::set-output name=job-id::6482431846'
2022-05-18T04:37:29.2094623Z Command completed after 1 attempt(s).
2022-05-18T04:37:29.2094894Z 
2022-05-18T04:37:29.2206509Z Prepare all required actions
2022-05-18T04:37:29.2206804Z Getting action download info
2022-05-18T04:37:29.3580504Z Download action repository 'actions/upload-artifact@v2' (SHA:82c141cc518b40d92cc801eee768e7aafc9c2fa2)
2022-05-18T04:37:29.4748509Z ##[group]Run ./.github/actions/upload-test-artifacts
2022-05-18T04:37:29.4748793Z with:
2022-05-18T04:37:29.4749031Z   file-suffix: test-distributed-1-1-linux.2xlarge_6482431846
2022-05-18T04:37:29.4749257Z env:
2022-05-18T04:37:29.4749400Z   IN_CI: 1
2022-05-18T04:37:29.4749559Z   IS_GHA: 1
2022-05-18T04:37:29.4749739Z   GIT_DEFAULT_BRANCH: master
2022-05-18T04:37:29.4749909Z ##[endgroup]
2022-05-18T04:37:29.4770633Z ##[group]Run # Remove any previous test jsons if they exist
2022-05-18T04:37:29.4770920Z # Remove any previous test jsons if they exist
2022-05-18T04:37:29.4771158Z rm -f test-jsons-*.zip
2022-05-18T04:37:29.4771390Z zip -r "test-jsons-${FILE_SUFFIX}.zip" test -i '*.json'
2022-05-18T04:37:29.4782288Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-05-18T04:37:29.4782506Z env:
2022-05-18T04:37:29.4782650Z   IN_CI: 1
2022-05-18T04:37:29.4782814Z   IS_GHA: 1
2022-05-18T04:37:29.4783209Z   GIT_DEFAULT_BRANCH: master
2022-05-18T04:37:29.4783457Z   FILE_SUFFIX: test-distributed-1-1-linux.2xlarge_6482431846
2022-05-18T04:37:29.4783692Z ##[endgroup]
2022-05-18T04:37:29.4903744Z   adding: test/allowlist_for_publicAPI.json (deflated 82%)
2022-05-18T04:37:29.4930013Z   adding: test/benchmark_utils/callgrind_artifacts.json (deflated 92%)
2022-05-18T04:37:29.4931007Z   adding: test/.pytorch-slow-tests.json (deflated 71%)
2022-05-18T04:37:29.4934195Z   adding: test/.pytorch-disabled-tests.json (deflated 83%)
2022-05-18T04:37:29.4950929Z ##[group]Run # Remove any previous test reports if they exist
2022-05-18T04:37:29.4951221Z # Remove any previous test reports if they exist
2022-05-18T04:37:29.4951445Z rm -f test-reports-*.zip
2022-05-18T04:37:29.4951704Z zip -r "test-reports-${FILE_SUFFIX}.zip" test -i '*.xml'
2022-05-18T04:37:29.4961717Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-05-18T04:37:29.4961918Z env:
2022-05-18T04:37:29.4962072Z   IN_CI: 1
2022-05-18T04:37:29.4962230Z   IS_GHA: 1
2022-05-18T04:37:29.4962396Z   GIT_DEFAULT_BRANCH: master
2022-05-18T04:37:29.4962647Z   FILE_SUFFIX: test-distributed-1-1-linux.2xlarge_6482431846
2022-05-18T04:37:29.4962879Z ##[endgroup]
2022-05-18T04:37:29.5133557Z   adding: test/test-reports/python-unittest/distributed._shard.checkpoint.test_checkpoint/TEST-TestStorageKeys-20220518033303.xml (deflated 40%)
2022-05-18T04:37:29.5134359Z   adding: test/test-reports/python-unittest/distributed._shard.checkpoint.test_checkpoint/TEST-TestDistributedCheckpointing-20220518033303.xml (deflated 81%)
2022-05-18T04:37:29.5135208Z   adding: test/test-reports/python-unittest/distributed._shard.checkpoint.test_file_system_checkpoint/TEST-TestDistributedStateDictSaveLoad-20220518033304.xml (deflated 42%)
2022-05-18T04:37:29.5136077Z   adding: test/test-reports/python-unittest/distributed._shard.checkpoint.test_file_system_checkpoint/TEST-TestDistributedReshardOnLoad-20220518033304.xml (deflated 69%)
2022-05-18T04:37:29.5137016Z   adding: test/test-reports/python-unittest/distributed._shard.checkpoint.test_file_system_checkpoint/TEST-TestDistributedStateDictSaveLoadWithSharedTensor-20220518033304.xml (deflated 45%)
2022-05-18T04:37:29.5137915Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_optim.test_sharded_optim/TEST-TestShardedOptimizer-20220518033305.xml (deflated 61%)
2022-05-18T04:37:29.5138724Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_binary_cmp/TEST-TestShardedTensorBinaryOps-20220518033306.xml (deflated 75%)
2022-05-18T04:37:29.5139488Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_chunk/TEST-TestShardedTensorChunkOps-20220518033306.xml (deflated 62%)
2022-05-18T04:37:29.5140549Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_elementwise_ops/TEST-TestShardedTensorElementWiseOps-20220518033307.xml (deflated 71%)
2022-05-18T04:37:29.5141371Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_embedding/TEST-TestShardedEmbedding-20220518033308.xml (deflated 62%)
2022-05-18T04:37:29.5142170Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_embedding_bag/TEST-TestShardedEmbeddingBag-20220518033308.xml (deflated 62%)
2022-05-18T04:37:29.5143083Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_init/TEST-TestShardedTensorNNInit-20220518033309.xml (deflated 71%)
2022-05-18T04:37:29.5143786Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_linear/TEST-TestShardedTensorOpsLinear-20220518033310.xml (deflated 71%)
2022-05-18T04:37:29.5144607Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_matrix_ops/TEST-TestShardedTensorMatrixOps-20220518033311.xml (deflated 88%)
2022-05-18T04:37:29.5145344Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_softmax/TEST-TestShardedSoftmax-20220518033312.xml (deflated 61%)
2022-05-18T04:37:29.5146039Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_tensor_ops/TEST-TestTensorOps-20220518033312.xml (deflated 74%)
2022-05-18T04:37:29.5146831Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.test_megatron_prototype/TEST-TestShardedTensorMegatronLinear-20220518033313.xml (deflated 44%)
2022-05-18T04:37:29.5147615Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardedTensorMetadata-20220518033314.xml (deflated 44%)
2022-05-18T04:37:29.5148417Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestCreateTensorFromParams-20220518033314.xml (deflated 44%)
2022-05-18T04:37:29.5149202Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestLocalTensor-20220518033314.xml (deflated 62%)
2022-05-18T04:37:29.5149958Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestModuleHookApi-20220518033314.xml (deflated 60%)
2022-05-18T04:37:29.5150660Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardParameter-20220518033314.xml (deflated 62%)
2022-05-18T04:37:29.5151315Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardTensor-20220518033314.xml (deflated 62%)
2022-05-18T04:37:29.5152048Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardedTensorChunked-20220518033314.xml (deflated 90%)
2022-05-18T04:37:29.5152867Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardedTensorCustomOps-20220518033314.xml (deflated 70%)
2022-05-18T04:37:29.5153627Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardedTensorEnumerable-20220518033314.xml (deflated 86%)
2022-05-18T04:37:29.5154465Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardedTensorFromLocalShards-20220518033314.xml (deflated 86%)
2022-05-18T04:37:29.5155281Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardedTensorFromLocalTensor-20220518033314.xml (deflated 63%)
2022-05-18T04:37:29.5156071Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor_reshard/TEST-TestReshard-20220518033316.xml (deflated 63%)
2022-05-18T04:37:29.5156974Z   adding: test/test-reports/python-unittest/distributed._shard.sharding_spec.test_sharding_spec/TEST-TestCustomShardingSpec-20220518033317.xml (deflated 67%)
2022-05-18T04:37:29.5157729Z   adding: test/test-reports/python-unittest/distributed._shard.sharding_spec.test_sharding_spec/TEST-TestShardingSpec-20220518033317.xml (deflated 78%)
2022-05-18T04:37:29.5158424Z   adding: test/test-reports/python-unittest/distributed._shard.test_partial_tensor/TEST-TestPartialTensorOps-20220518033319.xml (deflated 69%)
2022-05-18T04:37:29.5159130Z   adding: test/test-reports/python-unittest/distributed._shard.test_partial_tensor/TEST-TestPartialTensorReshard-20220518033319.xml (deflated 62%)
2022-05-18T04:37:29.5159786Z   adding: test/test-reports/python-unittest/distributed.algorithms.test_join/TEST-TestJoin-20220518033321.xml (deflated 80%)
2022-05-18T04:37:29.5160426Z   adding: test/test-reports/python-unittest/distributed.elastic.metrics.api_test/TEST-MetricsApiTest-20220518033330.xml (deflated 63%)
2022-05-18T04:37:29.5161186Z   adding: test/test-reports/python-unittest/distributed.elastic.multiprocessing.api_test/TEST-RunProcResultsTest-20220518033331.xml (deflated 56%)
2022-05-18T04:37:29.5161886Z   adding: test/test-reports/python-unittest/distributed.elastic.multiprocessing.api_test/TEST-StartProcessesListTest-20220518033331.xml (deflated 80%)
2022-05-18T04:37:29.5162648Z   adding: test/test-reports/python-unittest/distributed.elastic.multiprocessing.api_test/TEST-StartProcessesTest-20220518033331.xml (deflated 79%)
2022-05-18T04:37:29.5163388Z   adding: test/test-reports/python-unittest/distributed.elastic.multiprocessing.api_test/TEST-StdTest-20220518033331.xml (deflated 64%)
2022-05-18T04:37:29.5164073Z   adding: test/test-reports/python-unittest/distributed.elastic.timer.local_timer_example/TEST-LocalTimerExample-20220518033345.xml (deflated 54%)
2022-05-18T04:37:29.5164740Z   adding: test/test-reports/python-unittest/distributed.elastic.timer.local_timer_test/TEST-LocalTimerServerTest-20220518033352.xml (deflated 71%)
2022-05-18T04:37:29.5165397Z   adding: test/test-reports/python-unittest/distributed.elastic.timer.local_timer_test/TEST-LocalTimerTest-20220518033352.xml (deflated 69%)
2022-05-18T04:37:29.5166093Z   adding: test/test-reports/python-unittest/distributed.elastic.timer.local_timer_test/TEST-MultiprocessingRequestQueueTest-20220518033352.xml (deflated 66%)
2022-05-18T04:37:29.5166787Z   adding: test/test-reports/python-unittest/distributed.elastic.utils.distributed_test/TEST-DistributedUtilTest-20220518033356.xml (deflated 71%)
2022-05-18T04:37:29.5167425Z   adding: test/test-reports/python-unittest/distributed.elastic.utils.logging_test/TEST-LoggingTest-20220518033400.xml (deflated 55%)
2022-05-18T04:37:29.5168083Z   adding: test/test-reports/python-unittest/distributed.elastic.utils.util_test/TEST-StoreUtilTest-20220518033401.xml (deflated 63%)
2022-05-18T04:37:29.5168723Z   adding: test/test-reports/python-unittest/distributed.elastic.utils.util_test/TEST-UtilTest-20220518033401.xml (deflated 69%)
2022-05-18T04:37:29.5169409Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_distributed_checkpoint/TEST-TestDistributedCheckpoint-20220518033402.xml (deflated 62%)
2022-05-18T04:37:29.5170081Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_flatten_params_wrapper/TEST-TestFlattenParams-20220518033405.xml (deflated 81%)
2022-05-18T04:37:29.5170792Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_flatten_params_wrapper/TEST-TestFlattenParamsCUDA-20220518033405.xml (deflated 85%)
2022-05-18T04:37:29.5171567Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_flatten_params_wrapper/TEST-TestFlattenParamsCUDAHalf-20220518033405.xml (deflated 85%)
2022-05-18T04:37:29.5172224Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_apply/TEST-TestApply-20220518033406.xml (deflated 65%)
2022-05-18T04:37:29.5172870Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_checkpoint/TEST-TestFSDPCheckpoint-20220518033409.xml (deflated 84%)
2022-05-18T04:37:29.5173637Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_clip_grad_norm/TEST-TestCalcuGradNorm-20220518033414.xml (deflated 81%)
2022-05-18T04:37:29.5174274Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_clip_grad_norm/TEST-TestClipGradNorm-20220518033414.xml (deflated 87%)
2022-05-18T04:37:29.5174877Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_comm/TEST-TestCommunication-20220518033428.xml (deflated 91%)
2022-05-18T04:37:29.5175509Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_core/TEST-TestHooks-20220518033436.xml (deflated 82%)
2022-05-18T04:37:29.5176145Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_core/TEST-TestNoGrad-20220518033436.xml (deflated 58%)
2022-05-18T04:37:29.5176783Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_core/TEST-TestParamInit-20220518033436.xml (deflated 59%)
2022-05-18T04:37:29.5177383Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_core/TEST-TestParityWithDDP-20220518033436.xml (deflated 96%)
2022-05-18T04:37:29.5177987Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_exec_order/TEST-TestFSDPExecOrder-20220518033743.xml (deflated 84%)
2022-05-18T04:37:29.5178671Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_freezing_weights/TEST-TestFreezingWeights-20220518033752.xml (deflated 86%)
2022-05-18T04:37:29.5179376Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_grad_acc/TEST-TestGradAcc-20220518033800.xml (deflated 94%)
2022-05-18T04:37:29.5180063Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_ignored_modules/TEST-TestFSDPIgnoredModules-20220518033812.xml (deflated 66%)
2022-05-18T04:37:29.5180702Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_input/TEST-TestInput-20220518033816.xml (deflated 60%)
2022-05-18T04:37:29.5181331Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_memory/TEST-TestFSDPMemory-20220518033818.xml (deflated 59%)
2022-05-18T04:37:29.5181949Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_meta/TEST-TestFSDPWithMetaDevice-20220518033821.xml (deflated 88%)
2022-05-18T04:37:29.5182555Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_misc/TEST-TestFSDPMisc-20220518033828.xml (deflated 75%)
2022-05-18T04:37:29.5183339Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_mixed_precision/TEST-TestFSDPMixedPrecisionSharded-20220518033835.xml (deflated 95%)
2022-05-18T04:37:29.5184077Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_mixed_precision/TEST-TestFSDPMixedPrecisionUnsharded-20220518033835.xml (deflated 60%)
2022-05-18T04:37:29.5184756Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_multiple_forward/TEST-TestMultiForward-20220518033957.xml (deflated 42%)
2022-05-18T04:37:29.5185505Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_multiple_wrapping/TEST-TestMultipleWrapping-20220518033959.xml (deflated 46%)
2022-05-18T04:37:29.5186213Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_optim_state/TEST-TestFSDPOptimState-20220518034001.xml (deflated 91%)
2022-05-18T04:37:29.5186946Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_overlap/TEST-TestForwardOverlapWorldSizeOne-20220518034027.xml (deflated 44%)
2022-05-18T04:37:29.5187644Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_overlap/TEST-TestForwardOverlapWorldSizeTwo-20220518034027.xml (deflated 43%)
2022-05-18T04:37:29.5188334Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_pure_fp16/TEST-TestPureFP16-20220518034029.xml (deflated 52%)
2022-05-18T04:37:29.5189054Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_sharded_grad_scaler/TEST-TestShardGradScaler-20220518034031.xml (deflated 68%)
2022-05-18T04:37:29.5189928Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_sharded_grad_scaler/TEST-TestShardedGradScalerParityWithDDP-20220518034031.xml (deflated 85%)
2022-05-18T04:37:29.5190622Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_state_dict/TEST-TestFSDPStateDict-20220518034039.xml (deflated 95%)
2022-05-18T04:37:29.5191322Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_summon_full_params/TEST-TestSummonFullParams-20220518034126.xml (deflated 95%)
2022-05-18T04:37:29.5192016Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_summon_full_params/TEST-TestSummonFullParamsNoShard-20220518034126.xml (deflated 87%)
2022-05-18T04:37:29.5192733Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_traversal/TEST-TestTraversal-20220518034225.xml (deflated 43%)
2022-05-18T04:37:29.5193445Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_uneven/TEST-TestUnevenParamShard-20220518034227.xml (deflated 41%)
2022-05-18T04:37:29.5194109Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_utils/TEST-TestUtils-20220518034230.xml (deflated 68%)
2022-05-18T04:37:29.5194738Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_wrap/TEST-TestAutoWrap-20220518034231.xml (deflated 87%)
2022-05-18T04:37:29.5195388Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_wrap/TEST-TestFSDPWrap-20220518034231.xml (deflated 87%)
2022-05-18T04:37:29.5196050Z   adding: test/test-reports/python-unittest/distributed.nn.jit.test_instantiator/TEST-TestInstantiator-20220518034246.xml (deflated 64%)
2022-05-18T04:37:29.5196889Z   adding: test/test-reports/python-unittest/distributed.optim.test_zero_redundancy_optimizer/TEST-TestZeroRedundancyOptimizerDistributed-20220518034247.xml (deflated 91%)
2022-05-18T04:37:29.5197816Z   adding: test/test-reports/python-unittest/distributed.optim.test_zero_redundancy_optimizer/TEST-TestZeroRedundancyOptimizerSingleRank-20220518034247.xml (deflated 73%)
2022-05-18T04:37:29.5198676Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaDdpComparisonTest-20220518034338.xml (deflated 41%)
2022-05-18T04:37:29.5199507Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaDistAutogradTest-20220518034339.xml (deflated 42%)
2022-05-18T04:37:29.5200329Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaDistAutogradTest-20220518034341.xml (deflated 42%)
2022-05-18T04:37:29.5201150Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaDistAutogradTest-20220518034343.xml (deflated 42%)
2022-05-18T04:37:29.5201966Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaRemoteModuleTest-20220518034346.xml (deflated 42%)
2022-05-18T04:37:29.5202760Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaRemoteModuleTest-20220518034348.xml (deflated 42%)
2022-05-18T04:37:29.5203568Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaRemoteModuleTest-20220518034351.xml (deflated 42%)
2022-05-18T04:37:29.5204369Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaRemoteModuleTest-20220518034353.xml (deflated 42%)
2022-05-18T04:37:29.5205153Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaRpcTest-20220518034355.xml (deflated 41%)
2022-05-18T04:37:29.5205914Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20220518034358.xml (deflated 41%)
2022-05-18T04:37:29.5206720Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20220518034400.xml (deflated 41%)
2022-05-18T04:37:29.5207618Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20220518034402.xml (deflated 41%)
2022-05-18T04:37:29.5208437Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20220518034405.xml (deflated 41%)
2022-05-18T04:37:29.5209247Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20220518034407.xml (deflated 41%)
2022-05-18T04:37:29.5210026Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20220518034408.xml (deflated 41%)
2022-05-18T04:37:29.5210820Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20220518034409.xml (deflated 41%)
2022-05-18T04:37:29.5211600Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20220518034410.xml (deflated 41%)
2022-05-18T04:37:29.5212439Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518034410.xml (deflated 43%)
2022-05-18T04:37:29.5213317Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518034413.xml (deflated 43%)
2022-05-18T04:37:29.5214227Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518034416.xml (deflated 44%)
2022-05-18T04:37:29.5215089Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518034418.xml (deflated 44%)
2022-05-18T04:37:29.5215986Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518034420.xml (deflated 44%)
2022-05-18T04:37:29.5216878Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518034423.xml (deflated 44%)
2022-05-18T04:37:29.5217737Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518034425.xml (deflated 44%)
2022-05-18T04:37:29.5218601Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518034428.xml (deflated 44%)
2022-05-18T04:37:29.5219522Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518034430.xml (deflated 44%)
2022-05-18T04:37:29.5220387Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518034432.xml (deflated 45%)
2022-05-18T04:37:29.5221243Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518034435.xml (deflated 44%)
2022-05-18T04:37:29.5222159Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518034437.xml (deflated 44%)
2022-05-18T04:37:29.5223138Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518034440.xml (deflated 44%)
2022-05-18T04:37:29.5224375Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518034442.xml (deflated 44%)
2022-05-18T04:37:29.5225753Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518034445.xml (deflated 44%)
2022-05-18T04:37:29.5227111Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518034447.xml (deflated 44%)
2022-05-18T04:37:29.5228642Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518034450.xml (deflated 43%)
2022-05-18T04:37:29.5230081Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518034452.xml (deflated 43%)
2022-05-18T04:37:29.5231514Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518034455.xml (deflated 43%)
2022-05-18T04:37:29.5232886Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518034457.xml (deflated 43%)
2022-05-18T04:37:29.5234251Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518034500.xml (deflated 43%)
2022-05-18T04:37:29.5235637Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518034502.xml (deflated 43%)
2022-05-18T04:37:29.5237019Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518034505.xml (deflated 44%)
2022-05-18T04:37:29.5238376Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518034507.xml (deflated 44%)
2022-05-18T04:37:29.5239729Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518034510.xml (deflated 44%)
2022-05-18T04:37:29.5241099Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518034512.xml (deflated 44%)
2022-05-18T04:37:29.5242475Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518034515.xml (deflated 44%)
2022-05-18T04:37:29.5243848Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518034517.xml (deflated 44%)
2022-05-18T04:37:29.5245222Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518034519.xml (deflated 44%)
2022-05-18T04:37:29.5246569Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518034522.xml (deflated 44%)
2022-05-18T04:37:29.5247931Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518034524.xml (deflated 44%)
2022-05-18T04:37:29.5249287Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518034527.xml (deflated 43%)
2022-05-18T04:37:29.5250657Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518034529.xml (deflated 44%)
2022-05-18T04:37:29.5252008Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518034532.xml (deflated 44%)
2022-05-18T04:37:29.5253363Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518034534.xml (deflated 44%)
2022-05-18T04:37:29.5254724Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518034537.xml (deflated 44%)
2022-05-18T04:37:29.5256097Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518034539.xml (deflated 43%)
2022-05-18T04:37:29.5257584Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518034541.xml (deflated 44%)
2022-05-18T04:37:29.5258941Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518034544.xml (deflated 43%)
2022-05-18T04:37:29.5260323Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518034546.xml (deflated 43%)
2022-05-18T04:37:29.5261691Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518034549.xml (deflated 44%)
2022-05-18T04:37:29.5263151Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518034551.xml (deflated 44%)
2022-05-18T04:37:29.5264522Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518034553.xml (deflated 44%)
2022-05-18T04:37:29.5265906Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518034556.xml (deflated 43%)
2022-05-18T04:37:29.5267280Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518034558.xml (deflated 43%)
2022-05-18T04:37:29.5268650Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518034601.xml (deflated 44%)
2022-05-18T04:37:29.5270072Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518034603.xml (deflated 44%)
2022-05-18T04:37:29.5271417Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518034606.xml (deflated 44%)
2022-05-18T04:37:29.5272815Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518034608.xml (deflated 44%)
2022-05-18T04:37:29.5274188Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518034610.xml (deflated 44%)
2022-05-18T04:37:29.5275548Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518034613.xml (deflated 43%)
2022-05-18T04:37:29.5276907Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518034615.xml (deflated 43%)
2022-05-18T04:37:29.5278277Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518034618.xml (deflated 43%)
2022-05-18T04:37:29.5279648Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518034620.xml (deflated 44%)
2022-05-18T04:37:29.5281026Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518034623.xml (deflated 43%)
2022-05-18T04:37:29.5282387Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518034625.xml (deflated 43%)
2022-05-18T04:37:29.5283741Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518034628.xml (deflated 43%)
2022-05-18T04:37:29.5285107Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518034630.xml (deflated 43%)
2022-05-18T04:37:29.5286557Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518034633.xml (deflated 44%)
2022-05-18T04:37:29.5287990Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518034635.xml (deflated 44%)
2022-05-18T04:37:29.5289349Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518034637.xml (deflated 44%)
2022-05-18T04:37:29.5290728Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518034640.xml (deflated 44%)
2022-05-18T04:37:29.5292100Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518034642.xml (deflated 44%)
2022-05-18T04:37:29.5293495Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518034645.xml (deflated 44%)
2022-05-18T04:37:29.5294879Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518034647.xml (deflated 44%)
2022-05-18T04:37:29.5296242Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518034649.xml (deflated 44%)
2022-05-18T04:37:29.5297621Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518034652.xml (deflated 44%)
2022-05-18T04:37:29.5299001Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518034654.xml (deflated 44%)
2022-05-18T04:37:29.5300380Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518034657.xml (deflated 44%)
2022-05-18T04:37:29.5301741Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518034659.xml (deflated 43%)
2022-05-18T04:37:29.5303210Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518034702.xml (deflated 43%)
2022-05-18T04:37:29.5304416Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518034704.xml (deflated 43%)
2022-05-18T04:37:29.5305777Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518034707.xml (deflated 43%)
2022-05-18T04:37:29.5307124Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518034709.xml (deflated 43%)
2022-05-18T04:37:29.5308443Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518034711.xml (deflated 43%)
2022-05-18T04:37:29.5309862Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518034714.xml (deflated 43%)
2022-05-18T04:37:29.5311183Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518034716.xml (deflated 43%)
2022-05-18T04:37:29.5312515Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518034719.xml (deflated 43%)
2022-05-18T04:37:29.5313828Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518034721.xml (deflated 43%)
2022-05-18T04:37:29.5315153Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518034724.xml (deflated 43%)
2022-05-18T04:37:29.5316660Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518034726.xml (deflated 43%)
2022-05-18T04:37:29.5317987Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518034728.xml (deflated 43%)
2022-05-18T04:37:29.5319483Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518034731.xml (deflated 43%)
2022-05-18T04:37:29.5320834Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518034733.xml (deflated 43%)
2022-05-18T04:37:29.5322167Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518034736.xml (deflated 43%)
2022-05-18T04:37:29.5323495Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518034738.xml (deflated 43%)
2022-05-18T04:37:29.5324845Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518034741.xml (deflated 43%)
2022-05-18T04:37:29.5326168Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20220518034743.xml (deflated 44%)
2022-05-18T04:37:29.5327553Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeCudaDistAutogradTest-20220518034746.xml (deflated 44%)
2022-05-18T04:37:29.5328937Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeCudaDistAutogradTest-20220518034748.xml (deflated 44%)
2022-05-18T04:37:29.5330319Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeCudaDistAutogradTest-20220518034751.xml (deflated 44%)
2022-05-18T04:37:29.5331622Z   adding: test/test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentDistAutogradTest-20220518034754.xml (deflated 42%)
2022-05-18T04:37:29.5332879Z   adding: test/test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentDistAutogradTest-20220518034804.xml (deflated 42%)
2022-05-18T04:37:29.5334052Z   adding: test/test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20220518034812.xml (deflated 41%)
2022-05-18T04:37:29.5335162Z   adding: test/test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20220518034821.xml (deflated 41%)
2022-05-18T04:37:29.5336276Z   adding: test/test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20220518034829.xml (deflated 41%)
2022-05-18T04:37:29.5337422Z   adding: test/test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20220518034842.xml (deflated 42%)
2022-05-18T04:37:29.5338537Z   adding: test/test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20220518034851.xml (deflated 41%)
2022-05-18T04:37:29.5339637Z   adding: test/test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20220518034854.xml (deflated 42%)
2022-05-18T04:37:29.5340767Z   adding: test/test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20220518034902.xml (deflated 41%)
2022-05-18T04:37:29.5341882Z   adding: test/test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20220518034910.xml (deflated 41%)
2022-05-18T04:37:29.5343131Z   adding: test/test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20220518034915.xml (deflated 41%)
2022-05-18T04:37:29.5344387Z   adding: test/test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20220518034924.xml (deflated 41%)
2022-05-18T04:37:29.5345497Z   adding: test/test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20220518034932.xml (deflated 41%)
2022-05-18T04:37:29.5346607Z   adding: test/test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20220518034940.xml (deflated 41%)
2022-05-18T04:37:29.5347730Z   adding: test/test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20220518034944.xml (deflated 41%)
2022-05-18T04:37:29.5348900Z   adding: test/test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20220518035000.xml (deflated 41%)
2022-05-18T04:37:29.5349999Z   adding: test/test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20220518035011.xml (deflated 41%)
2022-05-18T04:37:29.5351140Z   adding: test/test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20220518035015.xml (deflated 41%)
2022-05-18T04:37:29.5352248Z   adding: test/test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20220518035024.xml (deflated 41%)
2022-05-18T04:37:29.5353349Z   adding: test/test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20220518035029.xml (deflated 41%)
2022-05-18T04:37:29.5354443Z   adding: test/test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20220518035038.xml (deflated 41%)
2022-05-18T04:37:29.5355561Z   adding: test/test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20220518035046.xml (deflated 41%)
2022-05-18T04:37:29.5356712Z   adding: test/test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyJitFaultyAgentRpcTest-20220518035055.xml (deflated 41%)
2022-05-18T04:37:29.5357887Z   adding: test/test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyJitFaultyAgentRpcTest-20220518035103.xml (deflated 41%)
2022-05-18T04:37:29.5359047Z   adding: test/test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyJitFaultyAgentRpcTest-20220518035112.xml (deflated 42%)
2022-05-18T04:37:29.5360207Z   adding: test/test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyJitFaultyAgentRpcTest-20220518035120.xml (deflated 41%)
2022-05-18T04:37:29.5361378Z   adding: test/test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyJitFaultyAgentRpcTest-20220518035125.xml (deflated 42%)
2022-05-18T04:37:29.5362524Z   adding: test/test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyJitFaultyAgentRpcTest-20220518035132.xml (deflated 42%)
2022-05-18T04:37:29.5363691Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDdpComparisonTest-20220518035140.xml (deflated 42%)
2022-05-18T04:37:29.5364911Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDdpComparisonTest-20220518035142.xml (deflated 42%)
2022-05-18T04:37:29.5366100Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDdpComparisonTest-20220518035143.xml (deflated 41%)
2022-05-18T04:37:29.5367298Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDdpComparisonTest-20220518035146.xml (deflated 42%)
2022-05-18T04:37:29.5368533Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDdpUnderDistAutogradTest-20220518035149.xml (deflated 42%)
2022-05-18T04:37:29.5369825Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDdpUnderDistAutogradTest-20220518035152.xml (deflated 41%)
2022-05-18T04:37:29.5371117Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDdpUnderDistAutogradTest-20220518035156.xml (deflated 41%)
2022-05-18T04:37:29.5372522Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDdpUnderDistAutogradTest-20220518035200.xml (deflated 42%)
2022-05-18T04:37:29.5373753Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220518035203.xml (deflated 43%)
2022-05-18T04:37:29.5374929Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220518035206.xml (deflated 41%)
2022-05-18T04:37:29.5376095Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220518035210.xml (deflated 41%)
2022-05-18T04:37:29.5377276Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220518035212.xml (deflated 41%)
2022-05-18T04:37:29.5378468Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220518035215.xml (deflated 40%)
2022-05-18T04:37:29.5379633Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220518035218.xml (deflated 40%)
2022-05-18T04:37:29.5380813Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220518035221.xml (deflated 42%)
2022-05-18T04:37:29.5382017Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220518035223.xml (deflated 41%)
2022-05-18T04:37:29.5383346Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220518035226.xml (deflated 42%)
2022-05-18T04:37:29.5384519Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220518035227.xml (deflated 40%)
2022-05-18T04:37:29.5385703Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220518035230.xml (deflated 41%)
2022-05-18T04:37:29.5386882Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220518035232.xml (deflated 41%)
2022-05-18T04:37:29.5388053Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220518035234.xml (deflated 41%)
2022-05-18T04:37:29.5389248Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220518035237.xml (deflated 41%)
2022-05-18T04:37:29.5390417Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220518035240.xml (deflated 40%)
2022-05-18T04:37:29.5391596Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220518035243.xml (deflated 40%)
2022-05-18T04:37:29.5392792Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220518035246.xml (deflated 40%)
2022-05-18T04:37:29.5393978Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220518035248.xml (deflated 40%)
2022-05-18T04:37:29.5395141Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220518035251.xml (deflated 40%)
2022-05-18T04:37:29.5396311Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220518035254.xml (deflated 40%)
2022-05-18T04:37:29.5397474Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220518035257.xml (deflated 41%)
2022-05-18T04:37:29.5398648Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220518035259.xml (deflated 40%)
2022-05-18T04:37:29.5399969Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220518035302.xml (deflated 40%)
2022-05-18T04:37:29.5401139Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220518035315.xml (deflated 42%)
2022-05-18T04:37:29.5402307Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220518035316.xml (deflated 40%)
2022-05-18T04:37:29.5403484Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220518035319.xml (deflated 40%)
2022-05-18T04:37:29.5404643Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220518035322.xml (deflated 40%)
2022-05-18T04:37:29.5405811Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220518035324.xml (deflated 40%)
2022-05-18T04:37:29.5407004Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220518035327.xml (deflated 49%)
2022-05-18T04:37:29.5408175Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220518035330.xml (deflated 41%)
2022-05-18T04:37:29.5409325Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220518035333.xml (deflated 41%)
2022-05-18T04:37:29.5410493Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220518035336.xml (deflated 41%)
2022-05-18T04:37:29.5411662Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220518035339.xml (deflated 41%)
2022-05-18T04:37:29.5412834Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220518035342.xml (deflated 41%)
2022-05-18T04:37:29.5414016Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220518035345.xml (deflated 42%)
2022-05-18T04:37:29.5415179Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220518035346.xml (deflated 41%)
2022-05-18T04:37:29.5416363Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220518035349.xml (deflated 41%)
2022-05-18T04:37:29.5417541Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220518035352.xml (deflated 41%)
2022-05-18T04:37:29.5418714Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220518035355.xml (deflated 41%)
2022-05-18T04:37:29.5419895Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220518035358.xml (deflated 41%)
2022-05-18T04:37:29.5421068Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220518035401.xml (deflated 40%)
2022-05-18T04:37:29.5422242Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220518035403.xml (deflated 41%)
2022-05-18T04:37:29.5423512Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220518035406.xml (deflated 40%)
2022-05-18T04:37:29.5424677Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220518035409.xml (deflated 40%)
2022-05-18T04:37:29.5425850Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220518035412.xml (deflated 41%)
2022-05-18T04:37:29.5427185Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220518035415.xml (deflated 40%)
2022-05-18T04:37:29.5428373Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220518035418.xml (deflated 40%)
2022-05-18T04:37:29.5429582Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220518035421.xml (deflated 41%)
2022-05-18T04:37:29.5430757Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220518035424.xml (deflated 41%)
2022-05-18T04:37:29.5431936Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220518035428.xml (deflated 41%)
2022-05-18T04:37:29.5433121Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220518035431.xml (deflated 41%)
2022-05-18T04:37:29.5434285Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220518035434.xml (deflated 41%)
2022-05-18T04:37:29.5435446Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220518035436.xml (deflated 43%)
2022-05-18T04:37:29.5436631Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220518035439.xml (deflated 41%)
2022-05-18T04:37:29.5437802Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220518035442.xml (deflated 41%)
2022-05-18T04:37:29.5438979Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220518035445.xml (deflated 41%)
2022-05-18T04:37:29.5440139Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220518035448.xml (deflated 40%)
2022-05-18T04:37:29.5441304Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220518035450.xml (deflated 41%)
2022-05-18T04:37:29.5442484Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220518035453.xml (deflated 41%)
2022-05-18T04:37:29.5443656Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220518035456.xml (deflated 40%)
2022-05-18T04:37:29.5444819Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220518035500.xml (deflated 42%)
2022-05-18T04:37:29.5445997Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220518035503.xml (deflated 41%)
2022-05-18T04:37:29.5447183Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistAutogradTest-20220518035506.xml (deflated 41%)
2022-05-18T04:37:29.5448372Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistOptimizerTest-20220518035508.xml (deflated 42%)
2022-05-18T04:37:29.5449677Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistOptimizerTest-20220518035510.xml (deflated 42%)
2022-05-18T04:37:29.5450864Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistOptimizerTest-20220518035513.xml (deflated 41%)
2022-05-18T04:37:29.5452057Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeDistOptimizerTest-20220518035515.xml (deflated 42%)
2022-05-18T04:37:29.5453260Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitDistAutogradTest-20220518035517.xml (deflated 41%)
2022-05-18T04:37:29.5454584Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitDistAutogradTest-20220518035519.xml (deflated 41%)
2022-05-18T04:37:29.5455791Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitDistAutogradTest-20220518035522.xml (deflated 41%)
2022-05-18T04:37:29.5457014Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitDistAutogradTest-20220518035525.xml (deflated 41%)
2022-05-18T04:37:29.5458167Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220518035528.xml (deflated 40%)
2022-05-18T04:37:29.5459287Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220518035531.xml (deflated 40%)
2022-05-18T04:37:29.5460389Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220518035534.xml (deflated 40%)
2022-05-18T04:37:29.5461515Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220518035537.xml (deflated 41%)
2022-05-18T04:37:29.5462620Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220518035539.xml (deflated 40%)
2022-05-18T04:37:29.5463830Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220518035542.xml (deflated 40%)
2022-05-18T04:37:29.5464906Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220518035545.xml (deflated 40%)
2022-05-18T04:37:29.5466006Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220518035548.xml (deflated 40%)
2022-05-18T04:37:29.5467119Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220518035551.xml (deflated 40%)
2022-05-18T04:37:29.5468235Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220518035553.xml (deflated 40%)
2022-05-18T04:37:29.5469379Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220518035556.xml (deflated 40%)
2022-05-18T04:37:29.5470468Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220518035559.xml (deflated 40%)
2022-05-18T04:37:29.5471574Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220518035602.xml (deflated 40%)
2022-05-18T04:37:29.5472683Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220518035604.xml (deflated 40%)
2022-05-18T04:37:29.5473772Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220518035607.xml (deflated 40%)
2022-05-18T04:37:29.5474905Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220518035610.xml (deflated 40%)
2022-05-18T04:37:29.5476010Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220518035613.xml (deflated 40%)
2022-05-18T04:37:29.5477104Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220518035615.xml (deflated 40%)
2022-05-18T04:37:29.5478209Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220518035618.xml (deflated 40%)
2022-05-18T04:37:29.5479284Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220518035621.xml (deflated 40%)
2022-05-18T04:37:29.5480391Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220518035624.xml (deflated 40%)
2022-05-18T04:37:29.5481643Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220518035627.xml (deflated 39%)
2022-05-18T04:37:29.5482757Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220518035629.xml (deflated 40%)
2022-05-18T04:37:29.5483848Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220518035632.xml (deflated 40%)
2022-05-18T04:37:29.5484959Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220518035635.xml (deflated 40%)
2022-05-18T04:37:29.5486075Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220518035638.xml (deflated 40%)
2022-05-18T04:37:29.5487169Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220518035641.xml (deflated 40%)
2022-05-18T04:37:29.5488270Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220518035644.xml (deflated 39%)
2022-05-18T04:37:29.5489375Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220518035646.xml (deflated 41%)
2022-05-18T04:37:29.5490475Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220518035649.xml (deflated 40%)
2022-05-18T04:37:29.5491580Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220518035652.xml (deflated 40%)
2022-05-18T04:37:29.5492690Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220518035655.xml (deflated 40%)
2022-05-18T04:37:29.5493665Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220518035658.xml (deflated 39%)
2022-05-18T04:37:29.5494795Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220518035701.xml (deflated 40%)
2022-05-18T04:37:29.5495890Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220518035704.xml (deflated 40%)
2022-05-18T04:37:29.5496974Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220518035707.xml (deflated 40%)
2022-05-18T04:37:29.5498077Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220518035710.xml (deflated 40%)
2022-05-18T04:37:29.5499176Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220518035712.xml (deflated 41%)
2022-05-18T04:37:29.5500274Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220518035715.xml (deflated 40%)
2022-05-18T04:37:29.5501373Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220518035718.xml (deflated 40%)
2022-05-18T04:37:29.5502456Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220518035721.xml (deflated 40%)
2022-05-18T04:37:29.5503644Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220518035724.xml (deflated 40%)
2022-05-18T04:37:29.5504741Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220518035727.xml (deflated 41%)
2022-05-18T04:37:29.5505843Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220518035729.xml (deflated 40%)
2022-05-18T04:37:29.5506920Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220518035732.xml (deflated 40%)
2022-05-18T04:37:29.5508189Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220518035735.xml (deflated 40%)
2022-05-18T04:37:29.5509370Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220518035738.xml (deflated 40%)
2022-05-18T04:37:29.5510468Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220518035741.xml (deflated 40%)
2022-05-18T04:37:29.5511553Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220518035744.xml (deflated 40%)
2022-05-18T04:37:29.5512654Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220518035747.xml (deflated 40%)
2022-05-18T04:37:29.5513765Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220518035749.xml (deflated 40%)
2022-05-18T04:37:29.5514886Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220518035752.xml (deflated 40%)
2022-05-18T04:37:29.5515961Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220518035755.xml (deflated 40%)
2022-05-18T04:37:29.5517067Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeJitRpcTest-20220518035758.xml (deflated 40%)
2022-05-18T04:37:29.5518239Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeParameterServerTest-20220518035801.xml (deflated 43%)
2022-05-18T04:37:29.5519523Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeReinforcementLearningRpcTest-20220518035803.xml (deflated 43%)
2022-05-18T04:37:29.5520771Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRemoteModuleTest-20220518035806.xml (deflated 41%)
2022-05-18T04:37:29.5521947Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRemoteModuleTest-20220518035809.xml (deflated 41%)
2022-05-18T04:37:29.5523125Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRemoteModuleTest-20220518035811.xml (deflated 41%)
2022-05-18T04:37:29.5524306Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRemoteModuleTest-20220518035814.xml (deflated 41%)
2022-05-18T04:37:29.5525485Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRemoteModuleTest-20220518035816.xml (deflated 41%)
2022-05-18T04:37:29.5526647Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRemoteModuleTest-20220518035819.xml (deflated 40%)
2022-05-18T04:37:29.5527843Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRemoteModuleTest-20220518035821.xml (deflated 41%)
2022-05-18T04:37:29.5529018Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRemoteModuleTest-20220518035824.xml (deflated 41%)
2022-05-18T04:37:29.5530213Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRemoteModuleTest-20220518035826.xml (deflated 41%)
2022-05-18T04:37:29.5531376Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRemoteModuleTest-20220518035829.xml (deflated 41%)
2022-05-18T04:37:29.5532539Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRemoteModuleTest-20220518035831.xml (deflated 42%)
2022-05-18T04:37:29.5533710Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRemoteModuleTest-20220518035834.xml (deflated 41%)
2022-05-18T04:37:29.5535024Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRemoteModuleTest-20220518035836.xml (deflated 41%)
2022-05-18T04:37:29.5536137Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518035839.xml (deflated 40%)
2022-05-18T04:37:29.5537222Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518035841.xml (deflated 40%)
2022-05-18T04:37:29.5538306Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518035844.xml (deflated 40%)
2022-05-18T04:37:29.5539386Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518035847.xml (deflated 40%)
2022-05-18T04:37:29.5540438Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518035850.xml (deflated 40%)
2022-05-18T04:37:29.5541517Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518035901.xml (deflated 40%)
2022-05-18T04:37:29.5542608Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518035904.xml (deflated 40%)
2022-05-18T04:37:29.5543781Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518035907.xml (deflated 40%)
2022-05-18T04:37:29.5544832Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518035910.xml (deflated 40%)
2022-05-18T04:37:29.5545902Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518035912.xml (deflated 40%)
2022-05-18T04:37:29.5546994Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518035915.xml (deflated 40%)
2022-05-18T04:37:29.5548068Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518035918.xml (deflated 40%)
2022-05-18T04:37:29.5549183Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518035921.xml (deflated 39%)
2022-05-18T04:37:29.5550269Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518035923.xml (deflated 39%)
2022-05-18T04:37:29.5551344Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518035926.xml (deflated 40%)
2022-05-18T04:37:29.5552412Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518035929.xml (deflated 40%)
2022-05-18T04:37:29.5553484Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518035932.xml (deflated 40%)
2022-05-18T04:37:29.5554556Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518035934.xml (deflated 40%)
2022-05-18T04:37:29.5555649Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518035937.xml (deflated 39%)
2022-05-18T04:37:29.5556719Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518035940.xml (deflated 40%)
2022-05-18T04:37:29.5557810Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518035942.xml (deflated 39%)
2022-05-18T04:37:29.5558866Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518035945.xml (deflated 40%)
2022-05-18T04:37:29.5559939Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518035948.xml (deflated 41%)
2022-05-18T04:37:29.5561127Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518035949.xml (deflated 40%)
2022-05-18T04:37:29.5562264Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518035952.xml (deflated 39%)
2022-05-18T04:37:29.5563332Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518035955.xml (deflated 39%)
2022-05-18T04:37:29.5564403Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518035957.xml (deflated 40%)
2022-05-18T04:37:29.5565480Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040000.xml (deflated 40%)
2022-05-18T04:37:29.5566547Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040003.xml (deflated 40%)
2022-05-18T04:37:29.5567602Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040006.xml (deflated 40%)
2022-05-18T04:37:29.5568681Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040009.xml (deflated 40%)
2022-05-18T04:37:29.5569763Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040011.xml (deflated 40%)
2022-05-18T04:37:29.5570821Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040015.xml (deflated 40%)
2022-05-18T04:37:29.5571870Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040019.xml (deflated 40%)
2022-05-18T04:37:29.5572944Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040022.xml (deflated 40%)
2022-05-18T04:37:29.5574007Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040024.xml (deflated 40%)
2022-05-18T04:37:29.5575094Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040027.xml (deflated 40%)
2022-05-18T04:37:29.5576146Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040029.xml (deflated 40%)
2022-05-18T04:37:29.5577226Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040032.xml (deflated 42%)
2022-05-18T04:37:29.5578306Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040035.xml (deflated 40%)
2022-05-18T04:37:29.5579373Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040038.xml (deflated 40%)
2022-05-18T04:37:29.5580448Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040041.xml (deflated 40%)
2022-05-18T04:37:29.5581522Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040043.xml (deflated 40%)
2022-05-18T04:37:29.5582597Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040046.xml (deflated 40%)
2022-05-18T04:37:29.5583763Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040049.xml (deflated 40%)
2022-05-18T04:37:29.5584843Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040052.xml (deflated 40%)
2022-05-18T04:37:29.5585895Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040055.xml (deflated 39%)
2022-05-18T04:37:29.5586973Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040057.xml (deflated 39%)
2022-05-18T04:37:29.5588189Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040100.xml (deflated 40%)
2022-05-18T04:37:29.5589807Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040103.xml (deflated 40%)
2022-05-18T04:37:29.5590716Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040107.xml (deflated 40%)
2022-05-18T04:37:29.5591800Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040110.xml (deflated 44%)
2022-05-18T04:37:29.5592887Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040115.xml (deflated 40%)
2022-05-18T04:37:29.5593965Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040118.xml (deflated 40%)
2022-05-18T04:37:29.5595045Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040121.xml (deflated 41%)
2022-05-18T04:37:29.5596124Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040123.xml (deflated 40%)
2022-05-18T04:37:29.5597178Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040126.xml (deflated 40%)
2022-05-18T04:37:29.5598240Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040128.xml (deflated 40%)
2022-05-18T04:37:29.5599287Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040131.xml (deflated 40%)
2022-05-18T04:37:29.5600373Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040134.xml (deflated 40%)
2022-05-18T04:37:29.5601439Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040137.xml (deflated 40%)
2022-05-18T04:37:29.5602496Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040139.xml (deflated 40%)
2022-05-18T04:37:29.5603563Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040142.xml (deflated 41%)
2022-05-18T04:37:29.5604612Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040143.xml (deflated 40%)
2022-05-18T04:37:29.5605707Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040146.xml (deflated 41%)
2022-05-18T04:37:29.5606790Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040149.xml (deflated 40%)
2022-05-18T04:37:29.5607874Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040152.xml (deflated 40%)
2022-05-18T04:37:29.5608921Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040155.xml (deflated 40%)
2022-05-18T04:37:29.5609998Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040157.xml (deflated 40%)
2022-05-18T04:37:29.5611082Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040200.xml (deflated 40%)
2022-05-18T04:37:29.5612148Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040203.xml (deflated 40%)
2022-05-18T04:37:29.5613210Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040206.xml (deflated 41%)
2022-05-18T04:37:29.5614361Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040207.xml (deflated 40%)
2022-05-18T04:37:29.5615493Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040210.xml (deflated 41%)
2022-05-18T04:37:29.5616567Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040213.xml (deflated 41%)
2022-05-18T04:37:29.5617623Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040215.xml (deflated 40%)
2022-05-18T04:37:29.5618703Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040218.xml (deflated 40%)
2022-05-18T04:37:29.5619783Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040220.xml (deflated 40%)
2022-05-18T04:37:29.5620853Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040223.xml (deflated 40%)
2022-05-18T04:37:29.5621911Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040226.xml (deflated 40%)
2022-05-18T04:37:29.5623074Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040229.xml (deflated 40%)
2022-05-18T04:37:29.5624110Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040232.xml (deflated 40%)
2022-05-18T04:37:29.5625178Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040234.xml (deflated 39%)
2022-05-18T04:37:29.5626247Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040237.xml (deflated 40%)
2022-05-18T04:37:29.5627333Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040240.xml (deflated 40%)
2022-05-18T04:37:29.5628418Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040243.xml (deflated 40%)
2022-05-18T04:37:29.5629570Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040246.xml (deflated 40%)
2022-05-18T04:37:29.5630646Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040248.xml (deflated 40%)
2022-05-18T04:37:29.5631697Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040251.xml (deflated 40%)
2022-05-18T04:37:29.5632771Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040254.xml (deflated 41%)
2022-05-18T04:37:29.5633851Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040257.xml (deflated 39%)
2022-05-18T04:37:29.5634934Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040300.xml (deflated 40%)
2022-05-18T04:37:29.5636003Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040302.xml (deflated 40%)
2022-05-18T04:37:29.5637102Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040305.xml (deflated 40%)
2022-05-18T04:37:29.5638177Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040308.xml (deflated 40%)
2022-05-18T04:37:29.5639255Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040311.xml (deflated 41%)
2022-05-18T04:37:29.5640412Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040313.xml (deflated 40%)
2022-05-18T04:37:29.5641555Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040316.xml (deflated 40%)
2022-05-18T04:37:29.5642640Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040319.xml (deflated 41%)
2022-05-18T04:37:29.5643716Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040321.xml (deflated 41%)
2022-05-18T04:37:29.5644774Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040324.xml (deflated 40%)
2022-05-18T04:37:29.5645842Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040327.xml (deflated 40%)
2022-05-18T04:37:29.5646925Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040330.xml (deflated 40%)
2022-05-18T04:37:29.5648015Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040333.xml (deflated 40%)
2022-05-18T04:37:29.5649076Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040335.xml (deflated 39%)
2022-05-18T04:37:29.5650153Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040338.xml (deflated 40%)
2022-05-18T04:37:29.5651288Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040344.xml (deflated 39%)
2022-05-18T04:37:29.5652387Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040350.xml (deflated 40%)
2022-05-18T04:37:29.5653429Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040353.xml (deflated 39%)
2022-05-18T04:37:29.5654492Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040355.xml (deflated 40%)
2022-05-18T04:37:29.5655571Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040358.xml (deflated 40%)
2022-05-18T04:37:29.5656650Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040401.xml (deflated 40%)
2022-05-18T04:37:29.5657708Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040407.xml (deflated 40%)
2022-05-18T04:37:29.5658781Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040413.xml (deflated 40%)
2022-05-18T04:37:29.5659878Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040416.xml (deflated 40%)
2022-05-18T04:37:29.5660955Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040418.xml (deflated 40%)
2022-05-18T04:37:29.5662023Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040421.xml (deflated 40%)
2022-05-18T04:37:29.5663200Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040424.xml (deflated 40%)
2022-05-18T04:37:29.5664284Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040427.xml (deflated 40%)
2022-05-18T04:37:29.5665371Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040429.xml (deflated 39%)
2022-05-18T04:37:29.5666434Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040432.xml (deflated 40%)
2022-05-18T04:37:29.5667650Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040435.xml (deflated 40%)
2022-05-18T04:37:29.5668766Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040440.xml (deflated 40%)
2022-05-18T04:37:29.5669868Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040444.xml (deflated 40%)
2022-05-18T04:37:29.5670945Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040447.xml (deflated 40%)
2022-05-18T04:37:29.5671996Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040450.xml (deflated 40%)
2022-05-18T04:37:29.5673075Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040453.xml (deflated 40%)
2022-05-18T04:37:29.5674163Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040455.xml (deflated 40%)
2022-05-18T04:37:29.5675229Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040458.xml (deflated 40%)
2022-05-18T04:37:29.5676286Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040501.xml (deflated 40%)
2022-05-18T04:37:29.5677358Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040504.xml (deflated 40%)
2022-05-18T04:37:29.5678432Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040506.xml (deflated 40%)
2022-05-18T04:37:29.5679512Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040509.xml (deflated 40%)
2022-05-18T04:37:29.5680571Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040512.xml (deflated 40%)
2022-05-18T04:37:29.5681630Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040515.xml (deflated 40%)
2022-05-18T04:37:29.5682695Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040518.xml (deflated 40%)
2022-05-18T04:37:29.5683760Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040520.xml (deflated 40%)
2022-05-18T04:37:29.5684806Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040523.xml (deflated 41%)
2022-05-18T04:37:29.5685882Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040526.xml (deflated 41%)
2022-05-18T04:37:29.5686959Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040529.xml (deflated 41%)
2022-05-18T04:37:29.5688036Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040532.xml (deflated 40%)
2022-05-18T04:37:29.5689078Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040534.xml (deflated 40%)
2022-05-18T04:37:29.5690150Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040537.xml (deflated 41%)
2022-05-18T04:37:29.5691223Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040540.xml (deflated 40%)
2022-05-18T04:37:29.5692299Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040542.xml (deflated 40%)
2022-05-18T04:37:29.5693439Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040545.xml (deflated 40%)
2022-05-18T04:37:29.5694561Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040548.xml (deflated 40%)
2022-05-18T04:37:29.5695641Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040551.xml (deflated 40%)
2022-05-18T04:37:29.5696716Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040554.xml (deflated 40%)
2022-05-18T04:37:29.5697785Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040556.xml (deflated 40%)
2022-05-18T04:37:29.5698836Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040559.xml (deflated 40%)
2022-05-18T04:37:29.5699918Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040602.xml (deflated 41%)
2022-05-18T04:37:29.5701005Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040605.xml (deflated 40%)
2022-05-18T04:37:29.5702069Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040607.xml (deflated 40%)
2022-05-18T04:37:29.5703188Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040610.xml (deflated 41%)
2022-05-18T04:37:29.5704295Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040613.xml (deflated 40%)
2022-05-18T04:37:29.5705380Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040617.xml (deflated 40%)
2022-05-18T04:37:29.5706471Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040621.xml (deflated 40%)
2022-05-18T04:37:29.5707558Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040623.xml (deflated 40%)
2022-05-18T04:37:29.5708650Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040626.xml (deflated 40%)
2022-05-18T04:37:29.5709812Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040635.xml (deflated 40%)
2022-05-18T04:37:29.5710909Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040638.xml (deflated 40%)
2022-05-18T04:37:29.5711793Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040641.xml (deflated 40%)
2022-05-18T04:37:29.5712411Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040644.xml (deflated 40%)
2022-05-18T04:37:29.5713030Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040647.xml (deflated 40%)
2022-05-18T04:37:29.5713644Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040649.xml (deflated 40%)
2022-05-18T04:37:29.5714244Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040652.xml (deflated 40%)
2022-05-18T04:37:29.5714855Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040655.xml (deflated 40%)
2022-05-18T04:37:29.5715468Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040657.xml (deflated 41%)
2022-05-18T04:37:29.5716076Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040700.xml (deflated 41%)
2022-05-18T04:37:29.5716783Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040703.xml (deflated 40%)
2022-05-18T04:37:29.5717397Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040706.xml (deflated 40%)
2022-05-18T04:37:29.5718016Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040709.xml (deflated 40%)
2022-05-18T04:37:29.5718629Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040713.xml (deflated 39%)
2022-05-18T04:37:29.5719383Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040716.xml (deflated 40%)
2022-05-18T04:37:29.5719956Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040719.xml (deflated 39%)
2022-05-18T04:37:29.5720537Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040722.xml (deflated 39%)
2022-05-18T04:37:29.5721118Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040724.xml (deflated 40%)
2022-05-18T04:37:29.5721697Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040728.xml (deflated 40%)
2022-05-18T04:37:29.5722261Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040731.xml (deflated 40%)
2022-05-18T04:37:29.5722834Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040734.xml (deflated 40%)
2022-05-18T04:37:29.5723414Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040737.xml (deflated 40%)
2022-05-18T04:37:29.5723996Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040740.xml (deflated 40%)
2022-05-18T04:37:29.5724553Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040742.xml (deflated 41%)
2022-05-18T04:37:29.5725126Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040745.xml (deflated 40%)
2022-05-18T04:37:29.5725701Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040749.xml (deflated 41%)
2022-05-18T04:37:29.5726283Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040752.xml (deflated 41%)
2022-05-18T04:37:29.5726846Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040755.xml (deflated 40%)
2022-05-18T04:37:29.5727419Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040757.xml (deflated 40%)
2022-05-18T04:37:29.5727995Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040800.xml (deflated 40%)
2022-05-18T04:37:29.5728567Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040803.xml (deflated 40%)
2022-05-18T04:37:29.5729138Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040806.xml (deflated 39%)
2022-05-18T04:37:29.5729698Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040810.xml (deflated 40%)
2022-05-18T04:37:29.5730276Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040813.xml (deflated 40%)
2022-05-18T04:37:29.5730945Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040816.xml (deflated 40%)
2022-05-18T04:37:29.5731523Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040819.xml (deflated 40%)
2022-05-18T04:37:29.5732082Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040822.xml (deflated 39%)
2022-05-18T04:37:29.5732661Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040825.xml (deflated 40%)
2022-05-18T04:37:29.5733234Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040828.xml (deflated 40%)
2022-05-18T04:37:29.5733809Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040831.xml (deflated 40%)
2022-05-18T04:37:29.5734375Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040834.xml (deflated 40%)
2022-05-18T04:37:29.5734969Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040836.xml (deflated 40%)
2022-05-18T04:37:29.5735544Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040840.xml (deflated 40%)
2022-05-18T04:37:29.5736103Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040843.xml (deflated 39%)
2022-05-18T04:37:29.5736680Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040846.xml (deflated 40%)
2022-05-18T04:37:29.5737253Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040849.xml (deflated 40%)
2022-05-18T04:37:29.5737833Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040852.xml (deflated 40%)
2022-05-18T04:37:29.5738395Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040855.xml (deflated 40%)
2022-05-18T04:37:29.5738971Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040859.xml (deflated 39%)
2022-05-18T04:37:29.5739547Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040902.xml (deflated 40%)
2022-05-18T04:37:29.5740121Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040905.xml (deflated 40%)
2022-05-18T04:37:29.5740680Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040908.xml (deflated 40%)
2022-05-18T04:37:29.5741259Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040912.xml (deflated 39%)
2022-05-18T04:37:29.5741835Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040916.xml (deflated 40%)
2022-05-18T04:37:29.5742411Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040919.xml (deflated 40%)
2022-05-18T04:37:29.5743059Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040922.xml (deflated 40%)
2022-05-18T04:37:29.5743642Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040925.xml (deflated 40%)
2022-05-18T04:37:29.5744218Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040928.xml (deflated 40%)
2022-05-18T04:37:29.5744839Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040931.xml (deflated 39%)
2022-05-18T04:37:29.5745423Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeRpcTest-20220518040934.xml (deflated 40%)
2022-05-18T04:37:29.5746094Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220518040936.xml (deflated 43%)
2022-05-18T04:37:29.5746849Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220518040939.xml (deflated 43%)
2022-05-18T04:37:29.5747602Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220518040942.xml (deflated 44%)
2022-05-18T04:37:29.5748352Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220518040945.xml (deflated 44%)
2022-05-18T04:37:29.5749154Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220518040948.xml (deflated 44%)
2022-05-18T04:37:29.5749908Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220518040951.xml (deflated 44%)
2022-05-18T04:37:29.5750660Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220518040954.xml (deflated 43%)
2022-05-18T04:37:29.5751405Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220518040957.xml (deflated 43%)
2022-05-18T04:37:29.5752141Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220518041000.xml (deflated 44%)
2022-05-18T04:37:29.5752891Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220518041003.xml (deflated 44%)
2022-05-18T04:37:29.5753643Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220518041006.xml (deflated 44%)
2022-05-18T04:37:29.5754389Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220518041008.xml (deflated 44%)
2022-05-18T04:37:29.5755120Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220518041011.xml (deflated 44%)
2022-05-18T04:37:29.5755861Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220518041014.xml (deflated 44%)
2022-05-18T04:37:29.5756601Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220518041017.xml (deflated 44%)
2022-05-18T04:37:29.5757345Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220518041020.xml (deflated 44%)
2022-05-18T04:37:29.5758198Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220518041024.xml (deflated 44%)
2022-05-18T04:37:29.5758928Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220518041027.xml (deflated 43%)
2022-05-18T04:37:29.5759667Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220518041030.xml (deflated 43%)
2022-05-18T04:37:29.5760409Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220518041033.xml (deflated 44%)
2022-05-18T04:37:29.5761213Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220518041036.xml (deflated 43%)
2022-05-18T04:37:29.5781136Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220518041039.xml (deflated 43%)
2022-05-18T04:37:29.5781905Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220518041042.xml (deflated 44%)
2022-05-18T04:37:29.5782659Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220518041045.xml (deflated 44%)
2022-05-18T04:37:29.5783504Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220518041048.xml (deflated 44%)
2022-05-18T04:37:29.5784260Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220518041052.xml (deflated 43%)
2022-05-18T04:37:29.5785015Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220518041055.xml (deflated 44%)
2022-05-18T04:37:29.5785767Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220518041058.xml (deflated 44%)
2022-05-18T04:37:29.5786506Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220518041101.xml (deflated 44%)
2022-05-18T04:37:29.5787241Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220518041104.xml (deflated 44%)
2022-05-18T04:37:29.5787984Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentDistAutogradTest-20220518041107.xml (deflated 44%)
2022-05-18T04:37:29.5788765Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220518041110.xml (deflated 43%)
2022-05-18T04:37:29.5789449Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220518041113.xml (deflated 43%)
2022-05-18T04:37:29.5790123Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220518041116.xml (deflated 43%)
2022-05-18T04:37:29.5790808Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220518041118.xml (deflated 44%)
2022-05-18T04:37:29.5791485Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220518041121.xml (deflated 43%)
2022-05-18T04:37:29.5792171Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220518041124.xml (deflated 44%)
2022-05-18T04:37:29.5792853Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220518041127.xml (deflated 43%)
2022-05-18T04:37:29.5793518Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220518041129.xml (deflated 43%)
2022-05-18T04:37:29.5794195Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220518041132.xml (deflated 43%)
2022-05-18T04:37:29.5794872Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220518041135.xml (deflated 43%)
2022-05-18T04:37:29.5795547Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220518041138.xml (deflated 43%)
2022-05-18T04:37:29.5796366Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220518041141.xml (deflated 43%)
2022-05-18T04:37:29.5797043Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220518041144.xml (deflated 43%)
2022-05-18T04:37:29.5797716Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220518041147.xml (deflated 43%)
2022-05-18T04:37:29.5798395Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220518041150.xml (deflated 43%)
2022-05-18T04:37:29.5799060Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220518041153.xml (deflated 43%)
2022-05-18T04:37:29.5799733Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220518041156.xml (deflated 43%)
2022-05-18T04:37:29.5800406Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220518041159.xml (deflated 43%)
2022-05-18T04:37:29.5801078Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220518041201.xml (deflated 43%)
2022-05-18T04:37:29.5801736Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220518041204.xml (deflated 44%)
2022-05-18T04:37:29.5802415Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220518041207.xml (deflated 43%)
2022-05-18T04:37:29.5803090Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220518041212.xml (deflated 43%)
2022-05-18T04:37:29.5803769Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220518041217.xml (deflated 43%)
2022-05-18T04:37:29.5804441Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220518041233.xml (deflated 43%)
2022-05-18T04:37:29.5805102Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220518041236.xml (deflated 44%)
2022-05-18T04:37:29.5805769Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220518041239.xml (deflated 43%)
2022-05-18T04:37:29.5806445Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220518041242.xml (deflated 44%)
2022-05-18T04:37:29.5807115Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220518041244.xml (deflated 43%)
2022-05-18T04:37:29.5807777Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220518041247.xml (deflated 43%)
2022-05-18T04:37:29.5808450Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220518041250.xml (deflated 43%)
2022-05-18T04:37:29.5809122Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220518041253.xml (deflated 43%)
2022-05-18T04:37:29.5809797Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220518041256.xml (deflated 44%)
2022-05-18T04:37:29.5810458Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220518041259.xml (deflated 44%)
2022-05-18T04:37:29.5811194Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220518041302.xml (deflated 43%)
2022-05-18T04:37:29.5811868Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220518041305.xml (deflated 43%)
2022-05-18T04:37:29.5812541Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentRpcTest-20220518041308.xml (deflated 43%)
2022-05-18T04:37:29.5813242Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeThreeWorkersRemoteModuleTest-20220518041310.xml (deflated 44%)
2022-05-18T04:37:29.5813957Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeThreeWorkersRemoteModuleTest-20220518041313.xml (deflated 43%)
2022-05-18T04:37:29.5814671Z   adding: test/test-reports/python-unittest/distributed.rpc.test_tensorpipe_agent/TEST-TensorPipeThreeWorkersRemoteModuleTest-20220518041315.xml (deflated 42%)
2022-05-18T04:37:29.5815282Z   adding: test/test-reports/python-unittest/distributed.test_c10d_common/TEST-CommTest-20220518041318.xml (deflated 38%)
2022-05-18T04:37:29.5815852Z   adding: test/test-reports/python-unittest/distributed.test_c10d_common/TEST-ComputeBucketAssignmentTest-20220518041320.xml (deflated 42%)
2022-05-18T04:37:29.5816459Z   adding: test/test-reports/python-unittest/distributed.test_c10d_common/TEST-ComputeBucketAssignmentTest-20220518041321.xml (deflated 41%)
2022-05-18T04:37:29.5817065Z   adding: test/test-reports/python-unittest/distributed.test_c10d_common/TEST-ComputeBucketAssignmentTest-20220518041322.xml (deflated 41%)
2022-05-18T04:37:29.5817667Z   adding: test/test-reports/python-unittest/distributed.test_c10d_common/TEST-ComputeBucketAssignmentTest-20220518041323.xml (deflated 41%)
2022-05-18T04:37:29.5818291Z   adding: test/test-reports/python-unittest/distributed.test_c10d_common/TEST-PythonProcessGroupExtensionTest-20220518041324.xml (deflated 41%)
2022-05-18T04:37:29.5818926Z   adding: test/test-reports/python-unittest/distributed.test_c10d_common/TEST-PythonProcessGroupExtensionTest-20220518041326.xml (deflated 41%)
2022-05-18T04:37:29.5819562Z   adding: test/test-reports/python-unittest/distributed.test_c10d_common/TEST-PythonProcessGroupExtensionTest-20220518041329.xml (deflated 41%)
2022-05-18T04:37:29.5820206Z   adding: test/test-reports/python-unittest/distributed.test_c10d_common/TEST-PythonProcessGroupExtensionTest-20220518041331.xml (deflated 41%)
2022-05-18T04:37:29.5820766Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20220518041335.xml (deflated 39%)
2022-05-18T04:37:29.5821253Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20220518041337.xml (deflated 40%)
2022-05-18T04:37:29.5821753Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20220518041338.xml (deflated 38%)
2022-05-18T04:37:29.5822247Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20220518041340.xml (deflated 39%)
2022-05-18T04:37:29.5822744Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20220518041342.xml (deflated 39%)
2022-05-18T04:37:29.5823322Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20220518041344.xml (deflated 39%)
2022-05-18T04:37:29.5823818Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20220518041346.xml (deflated 39%)
2022-05-18T04:37:29.5824304Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20220518041347.xml (deflated 39%)
2022-05-18T04:37:29.5824861Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220518041349.xml (deflated 44%)
2022-05-18T04:37:29.5825460Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220518041351.xml (deflated 45%)
2022-05-18T04:37:29.5826148Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220518041353.xml (deflated 43%)
2022-05-18T04:37:29.5826752Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220518041355.xml (deflated 43%)
2022-05-18T04:37:29.5827360Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220518041357.xml (deflated 45%)
2022-05-18T04:37:29.5827956Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220518041358.xml (deflated 45%)
2022-05-18T04:37:29.5828555Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220518041400.xml (deflated 46%)
2022-05-18T04:37:29.5829224Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220518041402.xml (deflated 46%)
2022-05-18T04:37:29.5829830Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220518041404.xml (deflated 44%)
2022-05-18T04:37:29.5830422Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220518041406.xml (deflated 45%)
2022-05-18T04:37:29.5831005Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220518041408.xml (deflated 45%)
2022-05-18T04:37:29.5831594Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220518041409.xml (deflated 44%)
2022-05-18T04:37:29.5832187Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220518041411.xml (deflated 44%)
2022-05-18T04:37:29.5832773Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220518041413.xml (deflated 44%)
2022-05-18T04:37:29.5833379Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220518041415.xml (deflated 43%)
2022-05-18T04:37:29.5833972Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220518041417.xml (deflated 45%)
2022-05-18T04:37:29.5834566Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220518041419.xml (deflated 44%)
2022-05-18T04:37:29.5835153Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220518041420.xml (deflated 46%)
2022-05-18T04:37:29.5835750Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220518041422.xml (deflated 45%)
2022-05-18T04:37:29.5836343Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220518041424.xml (deflated 50%)
2022-05-18T04:37:29.5836940Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220518041426.xml (deflated 42%)
2022-05-18T04:37:29.5837522Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220518041428.xml (deflated 42%)
2022-05-18T04:37:29.5838124Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220518041430.xml (deflated 42%)
2022-05-18T04:37:29.5838717Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220518041431.xml (deflated 42%)
2022-05-18T04:37:29.5839316Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220518041433.xml (deflated 43%)
2022-05-18T04:37:29.5839914Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220518041435.xml (deflated 42%)
2022-05-18T04:37:29.5840534Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220518041437.xml (deflated 42%)
2022-05-18T04:37:29.5841153Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220518041438.xml (deflated 42%)
2022-05-18T04:37:29.5841750Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220518041440.xml (deflated 42%)
2022-05-18T04:37:29.5842349Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220518041442.xml (deflated 45%)
2022-05-18T04:37:29.5842934Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220518041444.xml (deflated 46%)
2022-05-18T04:37:29.5843526Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220518041446.xml (deflated 41%)
2022-05-18T04:37:29.5844124Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220518041448.xml (deflated 42%)
2022-05-18T04:37:29.5844723Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220518041449.xml (deflated 42%)
2022-05-18T04:37:29.5845311Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220518041451.xml (deflated 42%)
2022-05-18T04:37:29.5845898Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220518041453.xml (deflated 42%)
2022-05-18T04:37:29.5846494Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20220518041455.xml (deflated 42%)
2022-05-18T04:37:29.5847072Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041457.xml (deflated 40%)
2022-05-18T04:37:29.5847626Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041459.xml (deflated 41%)
2022-05-18T04:37:29.5848198Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041501.xml (deflated 40%)
2022-05-18T04:37:29.5848759Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041503.xml (deflated 40%)
2022-05-18T04:37:29.5849314Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041505.xml (deflated 40%)
2022-05-18T04:37:29.5849852Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041507.xml (deflated 40%)
2022-05-18T04:37:29.5850416Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041508.xml (deflated 40%)
2022-05-18T04:37:29.5850972Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041512.xml (deflated 41%)
2022-05-18T04:37:29.5851525Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041514.xml (deflated 40%)
2022-05-18T04:37:29.5852067Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041516.xml (deflated 41%)
2022-05-18T04:37:29.5852621Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041518.xml (deflated 40%)
2022-05-18T04:37:29.5853173Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041520.xml (deflated 40%)
2022-05-18T04:37:29.5853722Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041522.xml (deflated 40%)
2022-05-18T04:37:29.5854267Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041524.xml (deflated 40%)
2022-05-18T04:37:29.5854854Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041526.xml (deflated 40%)
2022-05-18T04:37:29.5855432Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041528.xml (deflated 40%)
2022-05-18T04:37:29.5855984Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041530.xml (deflated 41%)
2022-05-18T04:37:29.5856531Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041532.xml (deflated 40%)
2022-05-18T04:37:29.5857087Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041535.xml (deflated 40%)
2022-05-18T04:37:29.5857637Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041537.xml (deflated 41%)
2022-05-18T04:37:29.5858193Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041539.xml (deflated 40%)
2022-05-18T04:37:29.5858737Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041541.xml (deflated 40%)
2022-05-18T04:37:29.5859290Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041543.xml (deflated 41%)
2022-05-18T04:37:29.5859841Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041545.xml (deflated 40%)
2022-05-18T04:37:29.5860399Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041547.xml (deflated 40%)
2022-05-18T04:37:29.5860945Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041550.xml (deflated 41%)
2022-05-18T04:37:29.5861499Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041552.xml (deflated 40%)
2022-05-18T04:37:29.5862054Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041554.xml (deflated 40%)
2022-05-18T04:37:29.5862610Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041556.xml (deflated 41%)
2022-05-18T04:37:29.5863245Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041558.xml (deflated 40%)
2022-05-18T04:37:29.5863800Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041559.xml (deflated 40%)
2022-05-18T04:37:29.5864356Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041601.xml (deflated 40%)
2022-05-18T04:37:29.5864909Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041605.xml (deflated 41%)
2022-05-18T04:37:29.5865451Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041607.xml (deflated 40%)
2022-05-18T04:37:29.5866008Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041609.xml (deflated 40%)
2022-05-18T04:37:29.5866569Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041611.xml (deflated 41%)
2022-05-18T04:37:29.5867125Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041613.xml (deflated 40%)
2022-05-18T04:37:29.5867663Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041615.xml (deflated 40%)
2022-05-18T04:37:29.5868217Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041619.xml (deflated 41%)
2022-05-18T04:37:29.5868816Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041621.xml (deflated 40%)
2022-05-18T04:37:29.5869432Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041623.xml (deflated 39%)
2022-05-18T04:37:29.5870012Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041625.xml (deflated 40%)
2022-05-18T04:37:29.5870572Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041627.xml (deflated 41%)
2022-05-18T04:37:29.5871125Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041629.xml (deflated 40%)
2022-05-18T04:37:29.5871674Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041631.xml (deflated 40%)
2022-05-18T04:37:29.5872212Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041635.xml (deflated 40%)
2022-05-18T04:37:29.5872769Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041637.xml (deflated 41%)
2022-05-18T04:37:29.5873332Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041638.xml (deflated 41%)
2022-05-18T04:37:29.5873885Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20220518041640.xml (deflated 40%)
2022-05-18T04:37:29.5874401Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ReducerTest-20220518041642.xml (deflated 39%)
2022-05-18T04:37:29.5874910Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ReducerTest-20220518041643.xml (deflated 39%)
2022-05-18T04:37:29.5875423Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ReducerTest-20220518041644.xml (deflated 39%)
2022-05-18T04:37:29.5875932Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ReducerTest-20220518041645.xml (deflated 39%)
2022-05-18T04:37:29.5876429Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ReducerTest-20220518041646.xml (deflated 40%)
2022-05-18T04:37:29.5876957Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-RendezvousEnvTest-20220518041646.xml (deflated 39%)
2022-05-18T04:37:29.5877481Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-TimeoutTest-20220518041647.xml (deflated 41%)
2022-05-18T04:37:29.5877982Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220518041649.xml (deflated 40%)
2022-05-18T04:37:29.5878463Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220518041650.xml (deflated 40%)
2022-05-18T04:37:29.5878957Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220518041651.xml (deflated 40%)
2022-05-18T04:37:29.5879448Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220518041652.xml (deflated 40%)
2022-05-18T04:37:29.5879941Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220518041653.xml (deflated 39%)
2022-05-18T04:37:29.5880417Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220518041654.xml (deflated 39%)
2022-05-18T04:37:29.5880907Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220518041655.xml (deflated 39%)
2022-05-18T04:37:29.5881395Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220518041656.xml (deflated 39%)
2022-05-18T04:37:29.5881886Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220518041657.xml (deflated 39%)
2022-05-18T04:37:29.5882364Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220518041658.xml (deflated 39%)
2022-05-18T04:37:29.5882851Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20220518041659.xml (deflated 39%)
2022-05-18T04:37:29.5883479Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518041700.xml (deflated 42%)
2022-05-18T04:37:29.5884091Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518041701.xml (deflated 42%)
2022-05-18T04:37:29.5884688Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518041702.xml (deflated 42%)
2022-05-18T04:37:29.5885291Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518041703.xml (deflated 43%)
2022-05-18T04:37:29.5885895Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518041704.xml (deflated 42%)
2022-05-18T04:37:29.5886498Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518041705.xml (deflated 44%)
2022-05-18T04:37:29.5887093Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518041707.xml (deflated 45%)
2022-05-18T04:37:29.5887691Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518041709.xml (deflated 43%)
2022-05-18T04:37:29.5888294Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518041711.xml (deflated 43%)
2022-05-18T04:37:29.5888890Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518041712.xml (deflated 45%)
2022-05-18T04:37:29.5889477Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518041714.xml (deflated 45%)
2022-05-18T04:37:29.5890070Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518041716.xml (deflated 46%)
2022-05-18T04:37:29.5890668Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518041718.xml (deflated 46%)
2022-05-18T04:37:29.5891270Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518041720.xml (deflated 44%)
2022-05-18T04:37:29.5891858Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518041721.xml (deflated 45%)
2022-05-18T04:37:29.5892453Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518041723.xml (deflated 45%)
2022-05-18T04:37:29.5893050Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518041725.xml (deflated 44%)
2022-05-18T04:37:29.5893646Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518041727.xml (deflated 44%)
2022-05-18T04:37:29.5894236Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518041729.xml (deflated 42%)
2022-05-18T04:37:29.5894832Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518041730.xml (deflated 42%)
2022-05-18T04:37:29.5895424Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518041731.xml (deflated 44%)
2022-05-18T04:37:29.5896023Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518041732.xml (deflated 42%)
2022-05-18T04:37:29.5896609Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518041733.xml (deflated 42%)
2022-05-18T04:37:29.5897202Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518041734.xml (deflated 43%)
2022-05-18T04:37:29.5897829Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518041735.xml (deflated 43%)
2022-05-18T04:37:29.5898452Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518041736.xml (deflated 42%)
2022-05-18T04:37:29.5899038Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518041737.xml (deflated 41%)
2022-05-18T04:37:29.5899631Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518041738.xml (deflated 41%)
2022-05-18T04:37:29.5900225Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518041739.xml (deflated 41%)
2022-05-18T04:37:29.5900821Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518041740.xml (deflated 43%)
2022-05-18T04:37:29.5901407Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518041741.xml (deflated 43%)
2022-05-18T04:37:29.5902007Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518041742.xml (deflated 42%)
2022-05-18T04:37:29.5902602Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518041743.xml (deflated 42%)
2022-05-18T04:37:29.5903283Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518041744.xml (deflated 41%)
2022-05-18T04:37:29.5903875Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518041746.xml (deflated 42%)
2022-05-18T04:37:29.5904469Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518041747.xml (deflated 42%)
2022-05-18T04:37:29.5905066Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518041748.xml (deflated 42%)
2022-05-18T04:37:29.5905665Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518041749.xml (deflated 42%)
2022-05-18T04:37:29.5906249Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518041750.xml (deflated 43%)
2022-05-18T04:37:29.5906850Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518041751.xml (deflated 43%)
2022-05-18T04:37:29.5907446Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518041752.xml (deflated 43%)
2022-05-18T04:37:29.5908040Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518041753.xml (deflated 44%)
2022-05-18T04:37:29.5908638Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518041754.xml (deflated 42%)
2022-05-18T04:37:29.5909295Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518041755.xml (deflated 41%)
2022-05-18T04:37:29.5909885Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518041756.xml (deflated 41%)
2022-05-18T04:37:29.5910481Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518041757.xml (deflated 42%)
2022-05-18T04:37:29.5911080Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20220518041758.xml (deflated 42%)
2022-05-18T04:37:29.5911650Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20220518041800.xml (deflated 41%)
2022-05-18T04:37:29.5912216Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20220518041801.xml (deflated 41%)
2022-05-18T04:37:29.5912866Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20220518041802.xml (deflated 41%)
2022-05-18T04:37:29.5913426Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20220518041803.xml (deflated 41%)
2022-05-18T04:37:29.5913977Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20220518041804.xml (deflated 42%)
2022-05-18T04:37:29.5914540Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20220518041805.xml (deflated 42%)
2022-05-18T04:37:29.5915100Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20220518041806.xml (deflated 42%)
2022-05-18T04:37:29.5915680Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLNoGPUTest-20220518041806.xml (deflated 42%)
2022-05-18T04:37:29.5916250Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220518041807.xml (deflated 42%)
2022-05-18T04:37:29.5916806Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220518041808.xml (deflated 42%)
2022-05-18T04:37:29.5917361Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220518041809.xml (deflated 41%)
2022-05-18T04:37:29.5917908Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220518041810.xml (deflated 42%)
2022-05-18T04:37:29.5918445Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220518041811.xml (deflated 42%)
2022-05-18T04:37:29.5919002Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220518041812.xml (deflated 42%)
2022-05-18T04:37:29.5919556Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220518041813.xml (deflated 42%)
2022-05-18T04:37:29.5920122Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220518041814.xml (deflated 41%)
2022-05-18T04:37:29.5920665Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220518041815.xml (deflated 41%)
2022-05-18T04:37:29.5921220Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220518041816.xml (deflated 41%)
2022-05-18T04:37:29.5921773Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220518041817.xml (deflated 41%)
2022-05-18T04:37:29.5922331Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20220518041818.xml (deflated 41%)
2022-05-18T04:37:29.5922864Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-RendezvousEnvTest-20220518041818.xml (deflated 41%)
2022-05-18T04:37:29.5923392Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-TimeoutTest-20220518041819.xml (deflated 40%)
2022-05-18T04:37:29.5924021Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-DistributedDataParallelSingleProcessTest-20220518041821.xml (deflated 44%)
2022-05-18T04:37:29.5924742Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-DistributedDataParallelSingleProcessTest-20220518041822.xml (deflated 44%)
2022-05-18T04:37:29.5925437Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-DistributedDataParallelSingleProcessTest-20220518041823.xml (deflated 44%)
2022-05-18T04:37:29.5926102Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-ProcessGroupShareTensorTest-20220518041824.xml (deflated 41%)
2022-05-18T04:37:29.5926729Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-ProcessGroupShareTensorTest-20220518041825.xml (deflated 42%)
2022-05-18T04:37:29.5927412Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-ProcessGroupShareTensorTest-20220518041826.xml (deflated 42%)
2022-05-18T04:37:29.5928025Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-ProcessGroupShareTensorTest-20220518041827.xml (deflated 42%)
2022-05-18T04:37:29.5928663Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20220518041828.xml (deflated 43%)
2022-05-18T04:37:29.5929304Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20220518041830.xml (deflated 44%)
2022-05-18T04:37:29.5929942Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20220518041832.xml (deflated 43%)
2022-05-18T04:37:29.5930566Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20220518041834.xml (deflated 43%)
2022-05-18T04:37:29.5931200Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20220518041836.xml (deflated 43%)
2022-05-18T04:37:29.5931830Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20220518041838.xml (deflated 43%)
2022-05-18T04:37:29.5932460Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20220518041841.xml (deflated 43%)
2022-05-18T04:37:29.5933093Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20220518041843.xml (deflated 43%)
2022-05-18T04:37:29.5933709Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-ProcessGroupShareTensorTest-20220518041846.xml (deflated 42%)
2022-05-18T04:37:29.5934333Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-ProcessGroupShareTensorTest-20220518041847.xml (deflated 42%)
2022-05-18T04:37:29.5934961Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-ProcessGroupShareTensorTest-20220518041848.xml (deflated 42%)
2022-05-18T04:37:29.5935579Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-ProcessGroupShareTensorTest-20220518041849.xml (deflated 42%)
2022-05-18T04:37:29.5936200Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20220518041850.xml (deflated 43%)
2022-05-18T04:37:29.5936830Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20220518041851.xml (deflated 43%)
2022-05-18T04:37:29.5937468Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20220518041852.xml (deflated 43%)
2022-05-18T04:37:29.5938099Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20220518041853.xml (deflated 43%)
2022-05-18T04:37:29.5938718Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20220518041854.xml (deflated 43%)
2022-05-18T04:37:29.5939348Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20220518041855.xml (deflated 44%)
2022-05-18T04:37:29.5939983Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20220518041856.xml (deflated 43%)
2022-05-18T04:37:29.5940573Z   adding: test/test-reports/python-unittest/distributed.test_data_parallel/TEST-TestDataParallel-20220518041857.xml (deflated 90%)
2022-05-18T04:37:29.5941165Z   adding: test/test-reports/python-unittest/distributed.test_data_parallel/TEST-TestDataParallelDeviceTypeCPU-20220518041857.xml (deflated 91%)
2022-05-18T04:37:29.5941794Z   adding: test/test-reports/python-unittest/distributed.test_launcher/TEST-TestDistributedLaunch-20220518043622.xml (deflated 46%)
2022-05-18T04:37:29.5942416Z   adding: test/test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20220518043625.xml (deflated 40%)
2022-05-18T04:37:29.5943192Z   adding: test/test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20220518043627.xml (deflated 40%)
2022-05-18T04:37:29.5943800Z   adding: test/test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20220518043630.xml (deflated 41%)
2022-05-18T04:37:29.5944412Z   adding: test/test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20220518043631.xml (deflated 41%)
2022-05-18T04:37:29.5945028Z   adding: test/test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20220518043633.xml (deflated 40%)
2022-05-18T04:37:29.5945641Z   adding: test/test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20220518043635.xml (deflated 40%)
2022-05-18T04:37:29.5946243Z   adding: test/test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20220518043638.xml (deflated 41%)
2022-05-18T04:37:29.5946845Z   adding: test/test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20220518043639.xml (deflated 41%)
2022-05-18T04:37:29.5947461Z   adding: test/test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20220518043641.xml (deflated 40%)
2022-05-18T04:37:29.5948068Z   adding: test/test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupNCCLWrapperTest-20220518043644.xml (deflated 41%)
2022-05-18T04:37:29.5948661Z   adding: test/test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupNCCLWrapperTest-20220518043645.xml (deflated 41%)
2022-05-18T04:37:29.5949328Z   adding: test/test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupNCCLWrapperTest-20220518043646.xml (deflated 41%)
2022-05-18T04:37:29.5949897Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-FileStoreTest-20220518043648.xml (deflated 40%)
2022-05-18T04:37:29.5950413Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-FileStoreTest-20220518043649.xml (deflated 40%)
2022-05-18T04:37:29.5950912Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-HashStoreTest-20220518043650.xml (deflated 40%)
2022-05-18T04:37:29.5951446Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-PrefixFileStoreTest-20220518043651.xml (deflated 40%)
2022-05-18T04:37:29.5951997Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-PrefixFileStoreTest-20220518043652.xml (deflated 40%)
2022-05-18T04:37:29.5952547Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-PrefixTCPStoreTest-20220518043653.xml (deflated 39%)
2022-05-18T04:37:29.5953081Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-PrefixTCPStoreTest-20220518043654.xml (deflated 39%)
2022-05-18T04:37:29.5953611Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-PythonStoreTest-20220518043655.xml (deflated 40%)
2022-05-18T04:37:29.5954145Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-RendezvousEnvTest-20220518043656.xml (deflated 39%)
2022-05-18T04:37:29.5954684Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-RendezvousFileTest-20220518043657.xml (deflated 40%)
2022-05-18T04:37:29.5955207Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-RendezvousFileTest-20220518043658.xml (deflated 39%)
2022-05-18T04:37:29.5955737Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-RendezvousTCPTest-20220518043659.xml (deflated 39%)
2022-05-18T04:37:29.5956273Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-RendezvousTCPTest-20220518043700.xml (deflated 40%)
2022-05-18T04:37:29.5956857Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-RendezvousTCPTest-20220518043701.xml (deflated 39%)
2022-05-18T04:37:29.5957406Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-RendezvousTCPTest-20220518043702.xml (deflated 40%)
2022-05-18T04:37:29.5957928Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-RendezvousTest-20220518043714.xml (deflated 39%)
2022-05-18T04:37:29.5958442Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20220518043715.xml (deflated 39%)
2022-05-18T04:37:29.5958948Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20220518043716.xml (deflated 39%)
2022-05-18T04:37:29.5959444Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20220518043717.xml (deflated 38%)
2022-05-18T04:37:29.5959947Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20220518043718.xml (deflated 38%)
2022-05-18T04:37:29.5960455Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20220518043719.xml (deflated 38%)
2022-05-18T04:37:29.5960957Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20220518043720.xml (deflated 39%)
2022-05-18T04:37:29.5961451Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20220518043721.xml (deflated 39%)
2022-05-18T04:37:29.5961954Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20220518043724.xml (deflated 39%)
2022-05-18T04:37:29.5962512Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041900.xml (deflated 43%)
2022-05-18T04:37:29.5963107Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041903.xml (deflated 42%)
2022-05-18T04:37:29.5963692Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041906.xml (deflated 42%)
2022-05-18T04:37:29.5964288Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041907.xml (deflated 41%)
2022-05-18T04:37:29.5964877Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041909.xml (deflated 41%)
2022-05-18T04:37:29.5965465Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041912.xml (deflated 41%)
2022-05-18T04:37:29.5966047Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041914.xml (deflated 41%)
2022-05-18T04:37:29.5966633Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041916.xml (deflated 41%)
2022-05-18T04:37:29.5967216Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041919.xml (deflated 41%)
2022-05-18T04:37:29.5967811Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041921.xml (deflated 41%)
2022-05-18T04:37:29.5968385Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041923.xml (deflated 41%)
2022-05-18T04:37:29.5968968Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041925.xml (deflated 41%)
2022-05-18T04:37:29.5969552Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041928.xml (deflated 42%)
2022-05-18T04:37:29.5970134Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041929.xml (deflated 41%)
2022-05-18T04:37:29.5970712Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041931.xml (deflated 42%)
2022-05-18T04:37:29.5971377Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041932.xml (deflated 43%)
2022-05-18T04:37:29.5971963Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041935.xml (deflated 42%)
2022-05-18T04:37:29.5972547Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041936.xml (deflated 46%)
2022-05-18T04:37:29.5973117Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041938.xml (deflated 47%)
2022-05-18T04:37:29.5973700Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041940.xml (deflated 48%)
2022-05-18T04:37:29.5974288Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041942.xml (deflated 46%)
2022-05-18T04:37:29.5974878Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041945.xml (deflated 40%)
2022-05-18T04:37:29.5975450Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041947.xml (deflated 40%)
2022-05-18T04:37:29.5976034Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041949.xml (deflated 41%)
2022-05-18T04:37:29.5976619Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041952.xml (deflated 40%)
2022-05-18T04:37:29.5977204Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041954.xml (deflated 41%)
2022-05-18T04:37:29.5977774Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041956.xml (deflated 40%)
2022-05-18T04:37:29.5978358Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518041959.xml (deflated 40%)
2022-05-18T04:37:29.5978943Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042001.xml (deflated 42%)
2022-05-18T04:37:29.5979523Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042002.xml (deflated 42%)
2022-05-18T04:37:29.5980097Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042003.xml (deflated 41%)
2022-05-18T04:37:29.5980679Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042005.xml (deflated 41%)
2022-05-18T04:37:29.5981259Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042008.xml (deflated 43%)
2022-05-18T04:37:29.5981843Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042009.xml (deflated 40%)
2022-05-18T04:37:29.5982415Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042012.xml (deflated 40%)
2022-05-18T04:37:29.5983085Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042014.xml (deflated 41%)
2022-05-18T04:37:29.5983665Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042017.xml (deflated 41%)
2022-05-18T04:37:29.5984238Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042019.xml (deflated 41%)
2022-05-18T04:37:29.5984815Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042021.xml (deflated 41%)
2022-05-18T04:37:29.5985387Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042024.xml (deflated 41%)
2022-05-18T04:37:29.5986059Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042026.xml (deflated 41%)
2022-05-18T04:37:29.5986639Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042028.xml (deflated 41%)
2022-05-18T04:37:29.5987219Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042031.xml (deflated 40%)
2022-05-18T04:37:29.5987791Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042033.xml (deflated 41%)
2022-05-18T04:37:29.5988370Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042036.xml (deflated 41%)
2022-05-18T04:37:29.5989021Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042038.xml (deflated 41%)
2022-05-18T04:37:29.5989607Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042040.xml (deflated 41%)
2022-05-18T04:37:29.5990178Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042043.xml (deflated 41%)
2022-05-18T04:37:29.5990750Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042045.xml (deflated 41%)
2022-05-18T04:37:29.5991328Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042047.xml (deflated 40%)
2022-05-18T04:37:29.5991905Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042050.xml (deflated 41%)
2022-05-18T04:37:29.5992470Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042052.xml (deflated 41%)
2022-05-18T04:37:29.5993058Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042055.xml (deflated 40%)
2022-05-18T04:37:29.5993637Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042057.xml (deflated 40%)
2022-05-18T04:37:29.5994214Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042059.xml (deflated 40%)
2022-05-18T04:37:29.5994783Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042102.xml (deflated 41%)
2022-05-18T04:37:29.5995365Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042104.xml (deflated 41%)
2022-05-18T04:37:29.5995950Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042107.xml (deflated 41%)
2022-05-18T04:37:29.5996539Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042109.xml (deflated 41%)
2022-05-18T04:37:29.5997112Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042112.xml (deflated 42%)
2022-05-18T04:37:29.5997695Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042114.xml (deflated 41%)
2022-05-18T04:37:29.5998279Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042116.xml (deflated 41%)
2022-05-18T04:37:29.5998857Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042119.xml (deflated 42%)
2022-05-18T04:37:29.5999429Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042121.xml (deflated 41%)
2022-05-18T04:37:29.6000044Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042123.xml (deflated 41%)
2022-05-18T04:37:29.6000651Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042126.xml (deflated 41%)
2022-05-18T04:37:29.6001234Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042128.xml (deflated 42%)
2022-05-18T04:37:29.6001804Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042130.xml (deflated 42%)
2022-05-18T04:37:29.6002385Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042132.xml (deflated 41%)
2022-05-18T04:37:29.6002963Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042135.xml (deflated 43%)
2022-05-18T04:37:29.6003550Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042136.xml (deflated 43%)
2022-05-18T04:37:29.6004123Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042137.xml (deflated 42%)
2022-05-18T04:37:29.6004705Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042138.xml (deflated 43%)
2022-05-18T04:37:29.6005033Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042139.xml (deflated 42%)
2022-05-18T04:37:29.6005361Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042140.xml (deflated 43%)
2022-05-18T04:37:29.6005690Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042141.xml (deflated 43%)
2022-05-18T04:37:29.6006024Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042142.xml (deflated 42%)
2022-05-18T04:37:29.6006355Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042143.xml (deflated 43%)
2022-05-18T04:37:29.6006684Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042144.xml (deflated 43%)
2022-05-18T04:37:29.6007000Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042145.xml (deflated 43%)
2022-05-18T04:37:29.6007327Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042146.xml (deflated 43%)
2022-05-18T04:37:29.6007657Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042147.xml (deflated 43%)
2022-05-18T04:37:29.6007985Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042148.xml (deflated 43%)
2022-05-18T04:37:29.6008320Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042149.xml (deflated 42%)
2022-05-18T04:37:29.6008648Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042150.xml (deflated 43%)
2022-05-18T04:37:29.6008977Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042151.xml (deflated 42%)
2022-05-18T04:37:29.6009304Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042152.xml (deflated 43%)
2022-05-18T04:37:29.6009628Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042153.xml (deflated 43%)
2022-05-18T04:37:29.6009957Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042154.xml (deflated 43%)
2022-05-18T04:37:29.6010335Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042155.xml (deflated 42%)
2022-05-18T04:37:29.6010658Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042157.xml (deflated 42%)
2022-05-18T04:37:29.6010985Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042158.xml (deflated 42%)
2022-05-18T04:37:29.6011312Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042159.xml (deflated 41%)
2022-05-18T04:37:29.6011640Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042202.xml (deflated 42%)
2022-05-18T04:37:29.6011969Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042205.xml (deflated 40%)
2022-05-18T04:37:29.6012301Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042208.xml (deflated 42%)
2022-05-18T04:37:29.6012631Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042210.xml (deflated 41%)
2022-05-18T04:37:29.6012963Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042214.xml (deflated 42%)
2022-05-18T04:37:29.6013285Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042216.xml (deflated 41%)
2022-05-18T04:37:29.6013616Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042219.xml (deflated 42%)
2022-05-18T04:37:29.6013930Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042220.xml (deflated 41%)
2022-05-18T04:37:29.6014265Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042227.xml (deflated 41%)
2022-05-18T04:37:29.6014594Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042230.xml (deflated 40%)
2022-05-18T04:37:29.6014922Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042232.xml (deflated 42%)
2022-05-18T04:37:29.6015253Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042233.xml (deflated 41%)
2022-05-18T04:37:29.6015580Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042234.xml (deflated 41%)
2022-05-18T04:37:29.6015909Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042235.xml (deflated 41%)
2022-05-18T04:37:29.6016239Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042236.xml (deflated 41%)
2022-05-18T04:37:29.6016568Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042237.xml (deflated 42%)
2022-05-18T04:37:29.6016897Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042238.xml (deflated 41%)
2022-05-18T04:37:29.6017225Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042239.xml (deflated 41%)
2022-05-18T04:37:29.6017540Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042241.xml (deflated 42%)
2022-05-18T04:37:29.6017866Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042243.xml (deflated 40%)
2022-05-18T04:37:29.6018193Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042246.xml (deflated 40%)
2022-05-18T04:37:29.6018575Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042249.xml (deflated 42%)
2022-05-18T04:37:29.6018905Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042251.xml (deflated 40%)
2022-05-18T04:37:29.6019232Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042253.xml (deflated 41%)
2022-05-18T04:37:29.6019560Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042254.xml (deflated 41%)
2022-05-18T04:37:29.6019890Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042255.xml (deflated 42%)
2022-05-18T04:37:29.6020216Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042257.xml (deflated 41%)
2022-05-18T04:37:29.6020553Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042259.xml (deflated 41%)
2022-05-18T04:37:29.6020869Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042302.xml (deflated 42%)
2022-05-18T04:37:29.6021198Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042303.xml (deflated 41%)
2022-05-18T04:37:29.6021525Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042304.xml (deflated 41%)
2022-05-18T04:37:29.6021853Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042306.xml (deflated 41%)
2022-05-18T04:37:29.6022180Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042308.xml (deflated 41%)
2022-05-18T04:37:29.6022510Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042309.xml (deflated 41%)
2022-05-18T04:37:29.6022838Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042310.xml (deflated 41%)
2022-05-18T04:37:29.6023257Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042312.xml (deflated 41%)
2022-05-18T04:37:29.6023583Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042313.xml (deflated 41%)
2022-05-18T04:37:29.6023911Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042314.xml (deflated 42%)
2022-05-18T04:37:29.6024225Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042316.xml (deflated 41%)
2022-05-18T04:37:29.6024556Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042317.xml (deflated 41%)
2022-05-18T04:37:29.6024885Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042320.xml (deflated 41%)
2022-05-18T04:37:29.6025209Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042322.xml (deflated 41%)
2022-05-18T04:37:29.6025538Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042324.xml (deflated 42%)
2022-05-18T04:37:29.6025867Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042327.xml (deflated 42%)
2022-05-18T04:37:29.6026192Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042329.xml (deflated 42%)
2022-05-18T04:37:29.6026575Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042331.xml (deflated 42%)
2022-05-18T04:37:29.6026933Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042333.xml (deflated 42%)
2022-05-18T04:37:29.6027265Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042336.xml (deflated 42%)
2022-05-18T04:37:29.6027596Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042338.xml (deflated 42%)
2022-05-18T04:37:29.6027913Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042340.xml (deflated 42%)
2022-05-18T04:37:29.6028241Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042343.xml (deflated 42%)
2022-05-18T04:37:29.6028568Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042345.xml (deflated 42%)
2022-05-18T04:37:29.6028950Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042347.xml (deflated 42%)
2022-05-18T04:37:29.6029279Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042350.xml (deflated 42%)
2022-05-18T04:37:29.6029610Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042352.xml (deflated 41%)
2022-05-18T04:37:29.6029934Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042353.xml (deflated 42%)
2022-05-18T04:37:29.6030259Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042355.xml (deflated 41%)
2022-05-18T04:37:29.6030588Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042357.xml (deflated 40%)
2022-05-18T04:37:29.6030918Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042400.xml (deflated 42%)
2022-05-18T04:37:29.6031233Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042402.xml (deflated 41%)
2022-05-18T04:37:29.6031562Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042403.xml (deflated 41%)
2022-05-18T04:37:29.6031888Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042404.xml (deflated 42%)
2022-05-18T04:37:29.6032214Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042405.xml (deflated 42%)
2022-05-18T04:37:29.6032543Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042406.xml (deflated 41%)
2022-05-18T04:37:29.6032875Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042407.xml (deflated 41%)
2022-05-18T04:37:29.6033202Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042409.xml (deflated 41%)
2022-05-18T04:37:29.6033528Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042411.xml (deflated 42%)
2022-05-18T04:37:29.6033857Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042412.xml (deflated 42%)
2022-05-18T04:37:29.6034185Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042413.xml (deflated 42%)
2022-05-18T04:37:29.6034513Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042415.xml (deflated 42%)
2022-05-18T04:37:29.6034885Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042417.xml (deflated 41%)
2022-05-18T04:37:29.6035214Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042418.xml (deflated 42%)
2022-05-18T04:37:29.6035537Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042419.xml (deflated 42%)
2022-05-18T04:37:29.6035865Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042421.xml (deflated 42%)
2022-05-18T04:37:29.6036193Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042424.xml (deflated 42%)
2022-05-18T04:37:29.6036522Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042426.xml (deflated 42%)
2022-05-18T04:37:29.6036850Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042428.xml (deflated 42%)
2022-05-18T04:37:29.6037179Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042429.xml (deflated 41%)
2022-05-18T04:37:29.6037510Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042432.xml (deflated 41%)
2022-05-18T04:37:29.6037840Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042435.xml (deflated 40%)
2022-05-18T04:37:29.6038158Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042437.xml (deflated 42%)
2022-05-18T04:37:29.6038487Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042440.xml (deflated 41%)
2022-05-18T04:37:29.6038819Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042443.xml (deflated 41%)
2022-05-18T04:37:29.6039148Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042445.xml (deflated 40%)
2022-05-18T04:37:29.6039474Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042447.xml (deflated 43%)
2022-05-18T04:37:29.6039804Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042448.xml (deflated 41%)
2022-05-18T04:37:29.6040133Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042451.xml (deflated 41%)
2022-05-18T04:37:29.6040460Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042453.xml (deflated 40%)
2022-05-18T04:37:29.6040791Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042455.xml (deflated 40%)
2022-05-18T04:37:29.6041122Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042458.xml (deflated 41%)
2022-05-18T04:37:29.6041444Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042500.xml (deflated 41%)
2022-05-18T04:37:29.6041760Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042502.xml (deflated 41%)
2022-05-18T04:37:29.6042087Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042505.xml (deflated 40%)
2022-05-18T04:37:29.6042415Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042507.xml (deflated 40%)
2022-05-18T04:37:29.6042740Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042510.xml (deflated 41%)
2022-05-18T04:37:29.6043114Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042513.xml (deflated 41%)
2022-05-18T04:37:29.6043444Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042515.xml (deflated 41%)
2022-05-18T04:37:29.6043770Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042518.xml (deflated 41%)
2022-05-18T04:37:29.6044095Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042520.xml (deflated 41%)
2022-05-18T04:37:29.6044422Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042521.xml (deflated 41%)
2022-05-18T04:37:29.6044750Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042522.xml (deflated 41%)
2022-05-18T04:37:29.6045068Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042526.xml (deflated 40%)
2022-05-18T04:37:29.6045395Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042531.xml (deflated 41%)
2022-05-18T04:37:29.6045720Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042533.xml (deflated 40%)
2022-05-18T04:37:29.6046049Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042535.xml (deflated 40%)
2022-05-18T04:37:29.6046376Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042538.xml (deflated 42%)
2022-05-18T04:37:29.6046704Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042539.xml (deflated 42%)
2022-05-18T04:37:29.6047036Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042540.xml (deflated 42%)
2022-05-18T04:37:29.6047362Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042541.xml (deflated 43%)
2022-05-18T04:37:29.6047691Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042542.xml (deflated 42%)
2022-05-18T04:37:29.6048019Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042543.xml (deflated 42%)
2022-05-18T04:37:29.6048338Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042544.xml (deflated 42%)
2022-05-18T04:37:29.6048664Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042545.xml (deflated 41%)
2022-05-18T04:37:29.6049172Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042547.xml (deflated 42%)
2022-05-18T04:37:29.6049503Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042549.xml (deflated 42%)
2022-05-18T04:37:29.6049835Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042550.xml (deflated 43%)
2022-05-18T04:37:29.6050160Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042551.xml (deflated 41%)
2022-05-18T04:37:29.6050485Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042553.xml (deflated 42%)
2022-05-18T04:37:29.6050814Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042556.xml (deflated 42%)
2022-05-18T04:37:29.6051171Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042558.xml (deflated 42%)
2022-05-18T04:37:29.6051532Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042600.xml (deflated 41%)
2022-05-18T04:37:29.6051866Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042601.xml (deflated 41%)
2022-05-18T04:37:29.6052179Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042603.xml (deflated 41%)
2022-05-18T04:37:29.6052511Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042605.xml (deflated 40%)
2022-05-18T04:37:29.6052836Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042607.xml (deflated 40%)
2022-05-18T04:37:29.6053163Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042609.xml (deflated 40%)
2022-05-18T04:37:29.6053498Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042612.xml (deflated 40%)
2022-05-18T04:37:29.6053826Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042614.xml (deflated 41%)
2022-05-18T04:37:29.6054153Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042617.xml (deflated 40%)
2022-05-18T04:37:29.6054479Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042619.xml (deflated 41%)
2022-05-18T04:37:29.6054808Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042621.xml (deflated 41%)
2022-05-18T04:37:29.6055135Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042624.xml (deflated 41%)
2022-05-18T04:37:29.6055467Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042626.xml (deflated 41%)
2022-05-18T04:37:29.6055785Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042629.xml (deflated 41%)
2022-05-18T04:37:29.6056112Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042631.xml (deflated 43%)
2022-05-18T04:37:29.6056437Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042632.xml (deflated 41%)
2022-05-18T04:37:29.6056766Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042634.xml (deflated 40%)
2022-05-18T04:37:29.6057095Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042637.xml (deflated 42%)
2022-05-18T04:37:29.6057425Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042638.xml (deflated 41%)
2022-05-18T04:37:29.6057752Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042641.xml (deflated 41%)
2022-05-18T04:37:29.6058078Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042643.xml (deflated 41%)
2022-05-18T04:37:29.6058408Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042645.xml (deflated 41%)
2022-05-18T04:37:29.6058733Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042648.xml (deflated 42%)
2022-05-18T04:37:29.6059049Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042649.xml (deflated 41%)
2022-05-18T04:37:29.6059442Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042650.xml (deflated 41%)
2022-05-18T04:37:29.6059773Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042652.xml (deflated 41%)
2022-05-18T04:37:29.6060101Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042654.xml (deflated 41%)
2022-05-18T04:37:29.6060428Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042657.xml (deflated 41%)
2022-05-18T04:37:29.6060761Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042659.xml (deflated 40%)
2022-05-18T04:37:29.6061087Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042701.xml (deflated 41%)
2022-05-18T04:37:29.6061421Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042704.xml (deflated 41%)
2022-05-18T04:37:29.6061752Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042706.xml (deflated 41%)
2022-05-18T04:37:29.6062080Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042709.xml (deflated 41%)
2022-05-18T04:37:29.6062405Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042710.xml (deflated 41%)
2022-05-18T04:37:29.6062723Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042711.xml (deflated 41%)
2022-05-18T04:37:29.6063138Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042713.xml (deflated 41%)
2022-05-18T04:37:29.6063467Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042716.xml (deflated 41%)
2022-05-18T04:37:29.6063795Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042718.xml (deflated 41%)
2022-05-18T04:37:29.6064122Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042720.xml (deflated 41%)
2022-05-18T04:37:29.6064446Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042723.xml (deflated 41%)
2022-05-18T04:37:29.6064771Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042725.xml (deflated 42%)
2022-05-18T04:37:29.6065098Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042727.xml (deflated 40%)
2022-05-18T04:37:29.6065428Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042730.xml (deflated 41%)
2022-05-18T04:37:29.6065758Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042732.xml (deflated 42%)
2022-05-18T04:37:29.6066075Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042734.xml (deflated 41%)
2022-05-18T04:37:29.6066400Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042735.xml (deflated 41%)
2022-05-18T04:37:29.6066727Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042736.xml (deflated 42%)
2022-05-18T04:37:29.6067053Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042739.xml (deflated 42%)
2022-05-18T04:37:29.6067381Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042740.xml (deflated 42%)
2022-05-18T04:37:29.6067788Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042742.xml (deflated 42%)
2022-05-18T04:37:29.6068119Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042743.xml (deflated 41%)
2022-05-18T04:37:29.6068444Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042746.xml (deflated 41%)
2022-05-18T04:37:29.6068844Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042748.xml (deflated 41%)
2022-05-18T04:37:29.6069180Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042751.xml (deflated 41%)
2022-05-18T04:37:29.6069493Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042753.xml (deflated 41%)
2022-05-18T04:37:29.6069827Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042755.xml (deflated 40%)
2022-05-18T04:37:29.6070156Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042757.xml (deflated 41%)
2022-05-18T04:37:29.6070480Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042800.xml (deflated 41%)
2022-05-18T04:37:29.6070806Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042802.xml (deflated 41%)
2022-05-18T04:37:29.6071130Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042804.xml (deflated 42%)
2022-05-18T04:37:29.6071460Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042805.xml (deflated 41%)
2022-05-18T04:37:29.6071791Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042808.xml (deflated 42%)
2022-05-18T04:37:29.6072118Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042809.xml (deflated 43%)
2022-05-18T04:37:29.6072446Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042811.xml (deflated 42%)
2022-05-18T04:37:29.6072770Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042812.xml (deflated 46%)
2022-05-18T04:37:29.6073085Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042815.xml (deflated 47%)
2022-05-18T04:37:29.6073413Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042817.xml (deflated 48%)
2022-05-18T04:37:29.6073743Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042819.xml (deflated 46%)
2022-05-18T04:37:29.6074071Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042821.xml (deflated 41%)
2022-05-18T04:37:29.6074398Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042824.xml (deflated 41%)
2022-05-18T04:37:29.6074727Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042826.xml (deflated 40%)
2022-05-18T04:37:29.6075054Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042828.xml (deflated 41%)
2022-05-18T04:37:29.6075380Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042831.xml (deflated 41%)
2022-05-18T04:37:29.6075736Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042833.xml (deflated 40%)
2022-05-18T04:37:29.6076086Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042835.xml (deflated 40%)
2022-05-18T04:37:29.6076401Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042838.xml (deflated 42%)
2022-05-18T04:37:29.6076732Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042839.xml (deflated 41%)
2022-05-18T04:37:29.6077061Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042840.xml (deflated 41%)
2022-05-18T04:37:29.6077393Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042842.xml (deflated 40%)
2022-05-18T04:37:29.6077721Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042844.xml (deflated 43%)
2022-05-18T04:37:29.6078051Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042845.xml (deflated 43%)
2022-05-18T04:37:29.6078377Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042846.xml (deflated 40%)
2022-05-18T04:37:29.6078703Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042848.xml (deflated 40%)
2022-05-18T04:37:29.6079031Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042851.xml (deflated 41%)
2022-05-18T04:37:29.6079360Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042853.xml (deflated 41%)
2022-05-18T04:37:29.6079685Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042856.xml (deflated 41%)
2022-05-18T04:37:29.6080008Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042858.xml (deflated 41%)
2022-05-18T04:37:29.6080338Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042901.xml (deflated 41%)
2022-05-18T04:37:29.6080665Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042903.xml (deflated 41%)
2022-05-18T04:37:29.6080990Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042906.xml (deflated 41%)
2022-05-18T04:37:29.6081317Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042908.xml (deflated 40%)
2022-05-18T04:37:29.6081643Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042910.xml (deflated 41%)
2022-05-18T04:37:29.6081978Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042913.xml (deflated 41%)
2022-05-18T04:37:29.6082303Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042915.xml (deflated 41%)
2022-05-18T04:37:29.6082628Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042918.xml (deflated 41%)
2022-05-18T04:37:29.6082953Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042920.xml (deflated 41%)
2022-05-18T04:37:29.6083267Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042922.xml (deflated 41%)
2022-05-18T04:37:29.6083598Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042925.xml (deflated 40%)
2022-05-18T04:37:29.6083975Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042927.xml (deflated 40%)
2022-05-18T04:37:29.6084309Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042930.xml (deflated 41%)
2022-05-18T04:37:29.6084636Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042932.xml (deflated 41%)
2022-05-18T04:37:29.6084963Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042934.xml (deflated 40%)
2022-05-18T04:37:29.6085288Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042937.xml (deflated 40%)
2022-05-18T04:37:29.6085612Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042939.xml (deflated 41%)
2022-05-18T04:37:29.6085943Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042942.xml (deflated 40%)
2022-05-18T04:37:29.6086270Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042944.xml (deflated 41%)
2022-05-18T04:37:29.6086597Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042946.xml (deflated 41%)
2022-05-18T04:37:29.6086911Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042949.xml (deflated 42%)
2022-05-18T04:37:29.6087240Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042951.xml (deflated 41%)
2022-05-18T04:37:29.6087566Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042953.xml (deflated 41%)
2022-05-18T04:37:29.6087894Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042956.xml (deflated 42%)
2022-05-18T04:37:29.6088223Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518042958.xml (deflated 41%)
2022-05-18T04:37:29.6088546Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043001.xml (deflated 41%)
2022-05-18T04:37:29.6088875Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043003.xml (deflated 41%)
2022-05-18T04:37:29.6089201Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043005.xml (deflated 42%)
2022-05-18T04:37:29.6089528Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043008.xml (deflated 41%)
2022-05-18T04:37:29.6089859Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043010.xml (deflated 41%)
2022-05-18T04:37:29.6090176Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043012.xml (deflated 43%)
2022-05-18T04:37:29.6090503Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043013.xml (deflated 43%)
2022-05-18T04:37:29.6090831Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043014.xml (deflated 42%)
2022-05-18T04:37:29.6091159Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043015.xml (deflated 42%)
2022-05-18T04:37:29.6091484Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043016.xml (deflated 42%)
2022-05-18T04:37:29.6091812Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043017.xml (deflated 43%)
2022-05-18T04:37:29.6092189Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043018.xml (deflated 42%)
2022-05-18T04:37:29.6092522Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043019.xml (deflated 43%)
2022-05-18T04:37:29.6092847Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043020.xml (deflated 43%)
2022-05-18T04:37:29.6093174Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043021.xml (deflated 43%)
2022-05-18T04:37:29.6093494Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043022.xml (deflated 43%)
2022-05-18T04:37:29.6093819Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043023.xml (deflated 43%)
2022-05-18T04:37:29.6094147Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043024.xml (deflated 43%)
2022-05-18T04:37:29.6094472Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043025.xml (deflated 43%)
2022-05-18T04:37:29.6094799Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043026.xml (deflated 43%)
2022-05-18T04:37:29.6095124Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043027.xml (deflated 43%)
2022-05-18T04:37:29.6095447Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043028.xml (deflated 42%)
2022-05-18T04:37:29.6095777Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043029.xml (deflated 42%)
2022-05-18T04:37:29.6096105Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043030.xml (deflated 43%)
2022-05-18T04:37:29.6096431Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043031.xml (deflated 43%)
2022-05-18T04:37:29.6096755Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043032.xml (deflated 42%)
2022-05-18T04:37:29.6097073Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043035.xml (deflated 42%)
2022-05-18T04:37:29.6097399Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043036.xml (deflated 41%)
2022-05-18T04:37:29.6097729Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043040.xml (deflated 42%)
2022-05-18T04:37:29.6098054Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043042.xml (deflated 40%)
2022-05-18T04:37:29.6098380Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043046.xml (deflated 41%)
2022-05-18T04:37:29.6098708Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043048.xml (deflated 41%)
2022-05-18T04:37:29.6099032Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043051.xml (deflated 42%)
2022-05-18T04:37:29.6099360Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043054.xml (deflated 40%)
2022-05-18T04:37:29.6099684Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043057.xml (deflated 40%)
2022-05-18T04:37:29.6100043Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043101.xml (deflated 40%)
2022-05-18T04:37:29.6100385Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043108.xml (deflated 41%)
2022-05-18T04:37:29.6100713Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043110.xml (deflated 41%)
2022-05-18T04:37:29.6101043Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043113.xml (deflated 41%)
2022-05-18T04:37:29.6101365Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043114.xml (deflated 41%)
2022-05-18T04:37:29.6101693Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043115.xml (deflated 41%)
2022-05-18T04:37:29.6102020Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043116.xml (deflated 41%)
2022-05-18T04:37:29.6102353Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043117.xml (deflated 41%)
2022-05-18T04:37:29.6102682Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043118.xml (deflated 41%)
2022-05-18T04:37:29.6103141Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043119.xml (deflated 41%)
2022-05-18T04:37:29.6103468Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043122.xml (deflated 42%)
2022-05-18T04:37:29.6103796Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043124.xml (deflated 40%)
2022-05-18T04:37:29.6104115Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043126.xml (deflated 41%)
2022-05-18T04:37:29.6104445Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043129.xml (deflated 42%)
2022-05-18T04:37:29.6104774Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043131.xml (deflated 40%)
2022-05-18T04:37:29.6105101Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043133.xml (deflated 41%)
2022-05-18T04:37:29.6105427Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043134.xml (deflated 41%)
2022-05-18T04:37:29.6105754Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043135.xml (deflated 42%)
2022-05-18T04:37:29.6106078Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043137.xml (deflated 41%)
2022-05-18T04:37:29.6106409Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043140.xml (deflated 42%)
2022-05-18T04:37:29.6106739Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043142.xml (deflated 42%)
2022-05-18T04:37:29.6107064Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043143.xml (deflated 41%)
2022-05-18T04:37:29.6107381Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043144.xml (deflated 41%)
2022-05-18T04:37:29.6107709Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043146.xml (deflated 41%)
2022-05-18T04:37:29.6108036Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043148.xml (deflated 41%)
2022-05-18T04:37:29.6108453Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043149.xml (deflated 41%)
2022-05-18T04:37:29.6108848Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043150.xml (deflated 41%)
2022-05-18T04:37:29.6109181Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043152.xml (deflated 41%)
2022-05-18T04:37:29.6109506Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043153.xml (deflated 41%)
2022-05-18T04:37:29.6109833Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043154.xml (deflated 42%)
2022-05-18T04:37:29.6110159Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043156.xml (deflated 41%)
2022-05-18T04:37:29.6110485Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043158.xml (deflated 41%)
2022-05-18T04:37:29.6110803Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043200.xml (deflated 41%)
2022-05-18T04:37:29.6111129Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043202.xml (deflated 41%)
2022-05-18T04:37:29.6111457Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043205.xml (deflated 42%)
2022-05-18T04:37:29.6111781Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043207.xml (deflated 42%)
2022-05-18T04:37:29.6112109Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043209.xml (deflated 42%)
2022-05-18T04:37:29.6112437Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043212.xml (deflated 42%)
2022-05-18T04:37:29.6112766Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043214.xml (deflated 42%)
2022-05-18T04:37:29.6113090Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043216.xml (deflated 41%)
2022-05-18T04:37:29.6113417Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043218.xml (deflated 42%)
2022-05-18T04:37:29.6113744Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043221.xml (deflated 42%)
2022-05-18T04:37:29.6114067Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043223.xml (deflated 42%)
2022-05-18T04:37:29.6114384Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043225.xml (deflated 42%)
2022-05-18T04:37:29.6114713Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043227.xml (deflated 42%)
2022-05-18T04:37:29.6115037Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043229.xml (deflated 42%)
2022-05-18T04:37:29.6115363Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043232.xml (deflated 41%)
2022-05-18T04:37:29.6115687Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043233.xml (deflated 42%)
2022-05-18T04:37:29.6116014Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043235.xml (deflated 41%)
2022-05-18T04:37:29.6116342Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043237.xml (deflated 40%)
2022-05-18T04:37:29.6116763Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043240.xml (deflated 42%)
2022-05-18T04:37:29.6117094Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043242.xml (deflated 41%)
2022-05-18T04:37:29.6117419Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043243.xml (deflated 41%)
2022-05-18T04:37:29.6117734Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043244.xml (deflated 42%)
2022-05-18T04:37:29.6118064Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043245.xml (deflated 41%)
2022-05-18T04:37:29.6118395Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043246.xml (deflated 42%)
2022-05-18T04:37:29.6118727Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043248.xml (deflated 41%)
2022-05-18T04:37:29.6119053Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043251.xml (deflated 42%)
2022-05-18T04:37:29.6119378Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043252.xml (deflated 42%)
2022-05-18T04:37:29.6119707Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043255.xml (deflated 41%)
2022-05-18T04:37:29.6120032Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043257.xml (deflated 41%)
2022-05-18T04:37:29.6120359Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043258.xml (deflated 42%)
2022-05-18T04:37:29.6120693Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043259.xml (deflated 41%)
2022-05-18T04:37:29.6121023Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043301.xml (deflated 42%)
2022-05-18T04:37:29.6121338Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043304.xml (deflated 42%)
2022-05-18T04:37:29.6121666Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043306.xml (deflated 41%)
2022-05-18T04:37:29.6121997Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043308.xml (deflated 42%)
2022-05-18T04:37:29.6122321Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043309.xml (deflated 41%)
2022-05-18T04:37:29.6122650Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043312.xml (deflated 40%)
2022-05-18T04:37:29.6122979Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043315.xml (deflated 40%)
2022-05-18T04:37:29.6123307Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043317.xml (deflated 42%)
2022-05-18T04:37:29.6123632Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043320.xml (deflated 40%)
2022-05-18T04:37:29.6123961Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043323.xml (deflated 41%)
2022-05-18T04:37:29.6124291Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043325.xml (deflated 41%)
2022-05-18T04:37:29.6124638Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043328.xml (deflated 40%)
2022-05-18T04:37:29.6124991Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043331.xml (deflated 40%)
2022-05-18T04:37:29.6125319Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043334.xml (deflated 40%)
2022-05-18T04:37:29.6125648Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043336.xml (deflated 40%)
2022-05-18T04:37:29.6125971Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043338.xml (deflated 41%)
2022-05-18T04:37:29.6126298Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043341.xml (deflated 40%)
2022-05-18T04:37:29.6126627Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043343.xml (deflated 41%)
2022-05-18T04:37:29.6126959Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043346.xml (deflated 40%)
2022-05-18T04:37:29.6127286Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043348.xml (deflated 40%)
2022-05-18T04:37:29.6127612Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043351.xml (deflated 41%)
2022-05-18T04:37:29.6127940Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043354.xml (deflated 41%)
2022-05-18T04:37:29.6128255Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043357.xml (deflated 41%)
2022-05-18T04:37:29.6128581Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043359.xml (deflated 41%)
2022-05-18T04:37:29.6128911Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043402.xml (deflated 41%)
2022-05-18T04:37:29.6129239Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043403.xml (deflated 42%)
2022-05-18T04:37:29.6129564Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043408.xml (deflated 41%)
2022-05-18T04:37:29.6129892Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043413.xml (deflated 40%)
2022-05-18T04:37:29.6130218Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043415.xml (deflated 40%)
2022-05-18T04:37:29.6130547Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043418.xml (deflated 40%)
2022-05-18T04:37:29.6130878Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043421.xml (deflated 42%)
2022-05-18T04:37:29.6131205Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043422.xml (deflated 42%)
2022-05-18T04:37:29.6131522Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043423.xml (deflated 42%)
2022-05-18T04:37:29.6131848Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043424.xml (deflated 43%)
2022-05-18T04:37:29.6132174Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043425.xml (deflated 42%)
2022-05-18T04:37:29.6132499Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043426.xml (deflated 42%)
2022-05-18T04:37:29.6132875Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043427.xml (deflated 42%)
2022-05-18T04:37:29.6133200Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043428.xml (deflated 41%)
2022-05-18T04:37:29.6133528Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043430.xml (deflated 42%)
2022-05-18T04:37:29.6133855Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043433.xml (deflated 42%)
2022-05-18T04:37:29.6134183Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043434.xml (deflated 43%)
2022-05-18T04:37:29.6134508Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043435.xml (deflated 41%)
2022-05-18T04:37:29.6134827Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043437.xml (deflated 41%)
2022-05-18T04:37:29.6135155Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043440.xml (deflated 42%)
2022-05-18T04:37:29.6135481Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043442.xml (deflated 41%)
2022-05-18T04:37:29.6135806Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043444.xml (deflated 41%)
2022-05-18T04:37:29.6136135Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043445.xml (deflated 41%)
2022-05-18T04:37:29.6136463Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043447.xml (deflated 41%)
2022-05-18T04:37:29.6136791Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043449.xml (deflated 40%)
2022-05-18T04:37:29.6137121Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043451.xml (deflated 40%)
2022-05-18T04:37:29.6137449Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043454.xml (deflated 40%)
2022-05-18T04:37:29.6137777Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043456.xml (deflated 40%)
2022-05-18T04:37:29.6138101Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043459.xml (deflated 41%)
2022-05-18T04:37:29.6138415Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043501.xml (deflated 41%)
2022-05-18T04:37:29.6138746Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043504.xml (deflated 41%)
2022-05-18T04:37:29.6139075Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043506.xml (deflated 41%)
2022-05-18T04:37:29.6139401Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043509.xml (deflated 41%)
2022-05-18T04:37:29.6139729Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043511.xml (deflated 41%)
2022-05-18T04:37:29.6140053Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043514.xml (deflated 41%)
2022-05-18T04:37:29.6140374Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043516.xml (deflated 43%)
2022-05-18T04:37:29.6140703Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043517.xml (deflated 41%)
2022-05-18T04:37:29.6141077Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043519.xml (deflated 41%)
2022-05-18T04:37:29.6141411Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043522.xml (deflated 42%)
2022-05-18T04:37:29.6141726Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043523.xml (deflated 41%)
2022-05-18T04:37:29.6142053Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043526.xml (deflated 41%)
2022-05-18T04:37:29.6142378Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043528.xml (deflated 41%)
2022-05-18T04:37:29.6142702Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043530.xml (deflated 41%)
2022-05-18T04:37:29.6143125Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043533.xml (deflated 41%)
2022-05-18T04:37:29.6143457Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043534.xml (deflated 41%)
2022-05-18T04:37:29.6143785Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043535.xml (deflated 40%)
2022-05-18T04:37:29.6144113Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043537.xml (deflated 40%)
2022-05-18T04:37:29.6144443Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043539.xml (deflated 40%)
2022-05-18T04:37:29.6144770Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043542.xml (deflated 41%)
2022-05-18T04:37:29.6145102Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043544.xml (deflated 40%)
2022-05-18T04:37:29.6145418Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043547.xml (deflated 41%)
2022-05-18T04:37:29.6145746Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043549.xml (deflated 41%)
2022-05-18T04:37:29.6146072Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043551.xml (deflated 41%)
2022-05-18T04:37:29.6146394Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043554.xml (deflated 42%)
2022-05-18T04:37:29.6146723Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043555.xml (deflated 41%)
2022-05-18T04:37:29.6147053Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043556.xml (deflated 41%)
2022-05-18T04:37:29.6147383Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043559.xml (deflated 40%)
2022-05-18T04:37:29.6147712Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043601.xml (deflated 41%)
2022-05-18T04:37:29.6148038Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043604.xml (deflated 41%)
2022-05-18T04:37:29.6148365Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043606.xml (deflated 40%)
2022-05-18T04:37:29.6148678Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043608.xml (deflated 41%)
2022-05-18T04:37:29.6149104Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043611.xml (deflated 42%)
2022-05-18T04:37:29.6149459Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043613.xml (deflated 40%)
2022-05-18T04:37:29.6149792Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043615.xml (deflated 41%)
2022-05-18T04:37:29.6150114Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043618.xml (deflated 42%)
2022-05-18T04:37:29.6150441Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043620.xml (deflated 41%)
2022-05-18T04:37:29.6150772Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20220518043621.xml (deflated 41%)
2022-05-18T04:37:29.6150981Z   adding: test/test-reports/cpp-rpc/test_rpc/test_cpp_rpc.xml (deflated 78%)
2022-05-18T04:37:29.6180083Z ##[group]Run seemethere/upload-artifact-s3@v4
2022-05-18T04:37:29.6180137Z with:
2022-05-18T04:37:29.6180221Z   retention-days: 14
2022-05-18T04:37:29.6180305Z   if-no-files-found: warn
2022-05-18T04:37:29.6180383Z   path: test-jsons-*.zip
2022-05-18T04:37:29.6180447Z   name: artifact
2022-05-18T04:37:29.6180525Z   s3-bucket: gha-artifacts
2022-05-18T04:37:29.6180595Z   region: us-east-1
2022-05-18T04:37:29.6180644Z env:
2022-05-18T04:37:29.6180703Z   IN_CI: 1
2022-05-18T04:37:29.6180765Z   IS_GHA: 1
2022-05-18T04:37:29.6180844Z   GIT_DEFAULT_BRANCH: master
2022-05-18T04:37:29.6180907Z ##[endgroup]
2022-05-18T04:37:29.9379688Z With the provided path, there will be 1 file uploaded
2022-05-18T04:37:29.9380223Z Uploading to s3 prefix: pytorch/pytorch/2342799944/1/artifact
2022-05-18T04:37:29.9387022Z Starting upload of test-jsons-test-distributed-1-1-linux.2xlarge_6482431846.zip
2022-05-18T04:37:30.0533537Z Finished upload of test-jsons-test-distributed-1-1-linux.2xlarge_6482431846.zip
2022-05-18T04:37:30.0625326Z ##[group]Run seemethere/upload-artifact-s3@v4
2022-05-18T04:37:30.0625539Z with:
2022-05-18T04:37:30.0625719Z   retention-days: 14
2022-05-18T04:37:30.0625901Z   if-no-files-found: error
2022-05-18T04:37:30.0626104Z   path: test-reports-*.zip
2022-05-18T04:37:30.0626289Z   name: artifact
2022-05-18T04:37:30.0626457Z   s3-bucket: gha-artifacts
2022-05-18T04:37:30.0626645Z   region: us-east-1
2022-05-18T04:37:30.0626815Z env:
2022-05-18T04:37:30.0626953Z   IN_CI: 1
2022-05-18T04:37:30.0627110Z   IS_GHA: 1
2022-05-18T04:37:30.0627288Z   GIT_DEFAULT_BRANCH: master
2022-05-18T04:37:30.0627458Z ##[endgroup]
2022-05-18T04:37:30.3823942Z With the provided path, there will be 1 file uploaded
2022-05-18T04:37:30.3824266Z Uploading to s3 prefix: pytorch/pytorch/2342799944/1/artifact
2022-05-18T04:37:30.3830633Z Starting upload of test-reports-test-distributed-1-1-linux.2xlarge_6482431846.zip
2022-05-18T04:37:30.5999227Z Finished upload of test-reports-test-distributed-1-1-linux.2xlarge_6482431846.zip
2022-05-18T04:37:30.6096243Z ##[group]Run set -x
2022-05-18T04:37:30.6096435Z set -x
2022-05-18T04:37:30.6096652Z python3 -m pip install -r requirements.txt
2022-05-18T04:37:30.6096897Z python3 -m pip install boto3==1.19.12
2022-05-18T04:37:30.6097184Z python3 -m tools.stats.print_test_stats --upload-to-s3 --compare-with-s3 test
2022-05-18T04:37:30.6108572Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-05-18T04:37:30.6108871Z env:
2022-05-18T04:37:30.6109027Z   IN_CI: 1
2022-05-18T04:37:30.6109172Z   IS_GHA: 1
2022-05-18T04:37:30.6109348Z   GIT_DEFAULT_BRANCH: master
2022-05-18T04:37:30.6109550Z   AWS_DEFAULT_REGION: us-east-1
2022-05-18T04:37:30.6109724Z   BRANCH: master
2022-05-18T04:37:30.6109941Z   JOB_BASE_NAME: linux-xenial-py3.7-gcc5.4-test
2022-05-18T04:37:30.6110165Z   TEST_CONFIG: distributed
2022-05-18T04:37:30.6110336Z   SHARD_NUMBER: 1
2022-05-18T04:37:30.6110632Z   BUILD_ENVIRONMENT: linux-xenial-py3.7-gcc5.4
2022-05-18T04:37:30.6110882Z   PR_NUMBER: 
2022-05-18T04:37:30.6111074Z   SHA1: 3b2375291aab7b48442f2e6fb1ef66cebc761e24
2022-05-18T04:37:30.6111268Z   TAG: 
2022-05-18T04:37:30.6111432Z   WORKFLOW_ID: 2342799944
2022-05-18T04:37:30.6111746Z   GITHUB_TOKEN: ***
2022-05-18T04:37:30.6111934Z   GHA_WORKFLOW_JOB_ID: 6482431846
2022-05-18T04:37:30.6112119Z ##[endgroup]
2022-05-18T04:37:30.6136063Z + python3 -m pip install -r requirements.txt
2022-05-18T04:37:30.8158545Z Defaulting to user installation because normal site-packages is not writeable
2022-05-18T04:37:30.8385821Z Ignoring dataclasses: markers 'python_version < "3.7"' don't match your environment
2022-05-18T04:37:30.8769072Z Collecting astunparse
2022-05-18T04:37:30.8907475Z   Downloading astunparse-1.6.3-py2.py3-none-any.whl (12 kB)
2022-05-18T04:37:30.9216810Z Collecting expecttest
2022-05-18T04:37:30.9270653Z   Downloading expecttest-0.1.3-py3-none-any.whl (6.5 kB)
2022-05-18T04:37:30.9530246Z Collecting future
2022-05-18T04:37:30.9560335Z   Downloading future-0.18.2.tar.gz (829 kB)
2022-05-18T04:37:31.7580687Z Collecting numpy
2022-05-18T04:37:31.7661509Z   Downloading numpy-1.21.6-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (15.7 MB)
2022-05-18T04:37:32.1429607Z Collecting psutil
2022-05-18T04:37:32.1466221Z   Downloading psutil-5.9.0-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (280 kB)
2022-05-18T04:37:32.2376520Z Collecting pyyaml
2022-05-18T04:37:32.2412557Z   Downloading PyYAML-6.0-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (596 kB)
2022-05-18T04:37:32.2562605Z Requirement already satisfied: requests in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 8)) (2.26.0)
2022-05-18T04:37:32.2677242Z Requirement already satisfied: setuptools in /usr/lib/python3.7/site-packages (from -r requirements.txt (line 9)) (49.1.3)
2022-05-18T04:37:32.2846402Z Requirement already satisfied: six in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 10)) (1.16.0)
2022-05-18T04:37:32.3029576Z Collecting types-dataclasses
2022-05-18T04:37:32.3062405Z   Downloading types_dataclasses-0.6.5-py3-none-any.whl (2.8 kB)
2022-05-18T04:37:32.3404388Z Collecting typing_extensions
2022-05-18T04:37:32.3434113Z   Downloading typing_extensions-4.2.0-py3-none-any.whl (24 kB)
2022-05-18T04:37:32.4017080Z Collecting wheel<1.0,>=0.23.0
2022-05-18T04:37:32.4047193Z   Downloading wheel-0.37.1-py2.py3-none-any.whl (35 kB)
2022-05-18T04:37:32.4138549Z Requirement already satisfied: urllib3<1.27,>=1.21.1 in /home/ec2-user/.local/lib/python3.7/site-packages (from requests->-r requirements.txt (line 8)) (1.26.9)
2022-05-18T04:37:32.4279138Z Requirement already satisfied: idna<4,>=2.5; python_version >= "3" in /home/ec2-user/.local/lib/python3.7/site-packages (from requests->-r requirements.txt (line 8)) (3.3)
2022-05-18T04:37:32.4289696Z Requirement already satisfied: charset-normalizer~=2.0.0; python_version >= "3" in /home/ec2-user/.local/lib/python3.7/site-packages (from requests->-r requirements.txt (line 8)) (2.0.12)
2022-05-18T04:37:32.4308833Z Requirement already satisfied: certifi>=2017.4.17 in /home/ec2-user/.local/lib/python3.7/site-packages (from requests->-r requirements.txt (line 8)) (2021.10.8)
2022-05-18T04:37:32.4315789Z Using legacy 'setup.py install' for future, since package 'wheel' is not installed.
2022-05-18T04:37:32.4620029Z Installing collected packages: wheel, astunparse, expecttest, future, numpy, psutil, pyyaml, types-dataclasses, typing-extensions
2022-05-18T04:37:32.4819861Z   WARNING: The script wheel is installed in '/home/ec2-user/.local/bin' which is not on PATH.
2022-05-18T04:37:32.4820339Z   Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
2022-05-18T04:37:32.5049124Z     Running setup.py install for future: started
2022-05-18T04:37:32.9656038Z     Running setup.py install for future: finished with status 'done'
2022-05-18T04:37:34.4792351Z   WARNING: The scripts f2py, f2py3 and f2py3.7 are installed in '/home/ec2-user/.local/bin' which is not on PATH.
2022-05-18T04:37:34.4792914Z   Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
2022-05-18T04:37:34.6645722Z Successfully installed astunparse-1.6.3 expecttest-0.1.3 future-0.18.2 numpy-1.21.6 psutil-5.9.0 pyyaml-6.0 types-dataclasses-0.6.5 typing-extensions-4.2.0 wheel-0.37.1
2022-05-18T04:37:34.7225368Z + python3 -m pip install boto3==1.19.12
2022-05-18T04:37:34.9252250Z Defaulting to user installation because normal site-packages is not writeable
2022-05-18T04:37:35.4711411Z Collecting boto3==1.19.12
2022-05-18T04:37:35.4849389Z   Downloading boto3-1.19.12-py3-none-any.whl (131 kB)
2022-05-18T04:37:35.4968589Z Requirement already satisfied: jmespath<1.0.0,>=0.7.1 in /home/ec2-user/.local/lib/python3.7/site-packages (from boto3==1.19.12) (0.10.0)
2022-05-18T04:37:36.1414317Z Collecting botocore<1.23.0,>=1.22.12
2022-05-18T04:37:36.1456976Z   Downloading botocore-1.22.12-py3-none-any.whl (8.1 MB)
2022-05-18T04:37:36.2803361Z Collecting s3transfer<0.6.0,>=0.5.0
2022-05-18T04:37:36.2835192Z   Downloading s3transfer-0.5.2-py3-none-any.whl (79 kB)
2022-05-18T04:37:36.2906825Z Requirement already satisfied: python-dateutil<3.0.0,>=2.1 in /home/ec2-user/.local/lib/python3.7/site-packages (from botocore<1.23.0,>=1.22.12->boto3==1.19.12) (2.8.2)
2022-05-18T04:37:36.2927960Z Requirement already satisfied: urllib3<1.27,>=1.25.4 in /home/ec2-user/.local/lib/python3.7/site-packages (from botocore<1.23.0,>=1.22.12->boto3==1.19.12) (1.26.9)
2022-05-18T04:37:36.3072421Z Requirement already satisfied: six>=1.5 in /home/ec2-user/.local/lib/python3.7/site-packages (from python-dateutil<3.0.0,>=2.1->botocore<1.23.0,>=1.22.12->boto3==1.19.12) (1.16.0)
2022-05-18T04:37:36.3679766Z Installing collected packages: botocore, s3transfer, boto3
2022-05-18T04:37:36.3680222Z   Attempting uninstall: botocore
2022-05-18T04:37:36.3683322Z     Found existing installation: botocore 1.19.63
2022-05-18T04:37:36.4240501Z     Uninstalling botocore-1.19.63:
2022-05-18T04:37:36.4372844Z       Successfully uninstalled botocore-1.19.63
2022-05-18T04:37:36.9471787Z   Attempting uninstall: s3transfer
2022-05-18T04:37:36.9473014Z     Found existing installation: s3transfer 0.3.7
2022-05-18T04:37:36.9511710Z     Uninstalling s3transfer-0.3.7:
2022-05-18T04:37:36.9516816Z       Successfully uninstalled s3transfer-0.3.7
2022-05-18T04:37:36.9882023Z   Attempting uninstall: boto3
2022-05-18T04:37:36.9883836Z     Found existing installation: boto3 1.16.34
2022-05-18T04:37:36.9983516Z     Uninstalling boto3-1.16.34:
2022-05-18T04:37:36.9998257Z       Successfully uninstalled boto3-1.16.34
2022-05-18T04:37:37.0501627Z Successfully installed boto3-1.19.12 botocore-1.22.12 s3transfer-0.5.2
2022-05-18T04:37:37.0935036Z + python3 -m tools.stats.print_test_stats --upload-to-s3 --compare-with-s3 test
2022-05-18T04:37:41.3651758Z [scribe] Scribe access token not provided, sending report via boto3...
2022-05-18T04:37:41.3652245Z 
2022-05-18T04:37:41.3652505Z ----- Historic stats comparison result ------
2022-05-18T04:37:41.3652676Z 
2022-05-18T04:37:41.3652839Z     job: linux-xenial-py3.7-gcc5.4-test
2022-05-18T04:37:41.3653090Z     commit: 3b2375291aab7b48442f2e6fb1ef66cebc761e24
2022-05-18T04:37:41.3653233Z 
2022-05-18T04:37:41.3653364Z Commit graph (base is most recent master ancestor with at least one S3 report):
2022-05-18T04:37:41.3653538Z 
2022-05-18T04:37:41.3653604Z     : (master)
2022-05-18T04:37:41.3653760Z     |
2022-05-18T04:37:41.3653948Z     * 3b2375291a (HEAD)              total time  2195.05s
2022-05-18T04:37:41.3656036Z     * 6e3391a7c3 (base)   3 reports, total time  1793.38s ± 1477.27s
2022-05-18T04:37:41.3656394Z     * 48581d74ad          3 reports, total time  1759.10s ± 1443.10s
2022-05-18T04:37:41.3656705Z     * c35bd8d423          3 reports, total time  1745.29s ± 1435.24s
2022-05-18T04:37:41.3657163Z     * f6beda89c6          4 reports, total time  1821.27s ± 1162.04s
2022-05-18T04:37:41.3657479Z     * ee080918df          4 reports, total time  1816.83s ± 1164.44s
2022-05-18T04:37:41.3657699Z     * bbaefdf6b5          0 reports
2022-05-18T04:37:41.3657877Z     * 7c52f204e0          0 reports
2022-05-18T04:37:41.3658059Z     * e0451d8022          0 reports
2022-05-18T04:37:41.3658340Z     * 4e2f5507d0          4 reports, total time  1878.04s ± 1210.75s
2022-05-18T04:37:41.3658651Z     * b64845eb18          4 reports, total time  1849.94s ± 1201.41s
2022-05-18T04:37:41.3658833Z     |
2022-05-18T04:37:41.3658981Z     :
2022-05-18T04:37:41.3659070Z 
2022-05-18T04:37:41.3659186Z Removed  (across  616 suites)     0 tests, totaling     0.00s
2022-05-18T04:37:41.3659428Z Modified (across    0 suites)     0 tests, totaling     0.00s
2022-05-18T04:37:41.3659679Z Added    (across  145 suites)  2019 tests, totaling +2471.40s
2022-05-18T04:37:41.4167633Z Prepare all required actions
2022-05-18T04:37:41.4186085Z ##[group]Run ./.github/actions/teardown-linux
2022-05-18T04:37:41.4186353Z with:
2022-05-18T04:37:41.4186576Z env:
2022-05-18T04:37:41.4186739Z   IN_CI: 1
2022-05-18T04:37:41.4186970Z   IS_GHA: 1
2022-05-18T04:37:41.4187210Z   GIT_DEFAULT_BRANCH: master
2022-05-18T04:37:41.4187403Z ##[endgroup]
2022-05-18T04:37:41.4221442Z ##[group]Run .github/scripts/wait_for_ssh_to_drain.sh
2022-05-18T04:37:41.4221705Z .github/scripts/wait_for_ssh_to_drain.sh
2022-05-18T04:37:41.4233036Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-05-18T04:37:41.4233264Z env:
2022-05-18T04:37:41.4233432Z   IN_CI: 1
2022-05-18T04:37:41.4233587Z   IS_GHA: 1
2022-05-18T04:37:41.4233780Z   GIT_DEFAULT_BRANCH: master
2022-05-18T04:37:41.4233979Z ##[endgroup]
2022-05-18T04:37:41.4275048Z Holding runner for 2 hours until all ssh sessions have logged out
2022-05-18T04:37:41.4312581Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty
2022-05-18T04:37:41.4312897Z # ignore expansion of "docker ps -q" since it could be empty
2022-05-18T04:37:41.4313143Z # shellcheck disable=SC2046
2022-05-18T04:37:41.4313432Z docker stop $(docker ps -q) || true
2022-05-18T04:37:41.4313703Z # Prune all of the docker images
2022-05-18T04:37:41.4313914Z docker system prune -af
2022-05-18T04:37:41.4326902Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-05-18T04:37:41.4327122Z env:
2022-05-18T04:37:41.4327265Z   IN_CI: 1
2022-05-18T04:37:41.4327425Z   IS_GHA: 1
2022-05-18T04:37:41.4327603Z   GIT_DEFAULT_BRANCH: master
2022-05-18T04:37:41.4327790Z ##[endgroup]
2022-05-18T04:37:42.0202610Z e67028644d5a
2022-05-18T04:37:42.3610696Z Deleted Containers:
2022-05-18T04:37:42.3611099Z e67028644d5a6a7816d5dabbfe01034cc7b659531e936f1e41c2e206a19b4065
2022-05-18T04:37:42.3611325Z 
2022-05-18T04:37:44.7079340Z Deleted Images:
2022-05-18T04:37:44.7080090Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-xenial-py3.7-gcc5.4:6deab82db6a72ca54cd3e3322ee4f13864536734
2022-05-18T04:37:44.7080795Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-xenial-py3.7-gcc5.4@sha256:9c228d64aeaa1a84153f684d8bf8d2b818b53df05ec50809bfb8bb625f2aea5c
2022-05-18T04:37:44.7081297Z deleted: sha256:59de092f48b8a69bedc5a97cdf7eb5f359b81a9ab8db3a062ddf64b0eeb3218c
2022-05-18T04:37:44.7081635Z deleted: sha256:a5c218bd38a05a6a5d34cd5b8705d6ef377789e4ad88a0452aff96d9d15ba536
2022-05-18T04:37:44.7081985Z deleted: sha256:98b641aa9ac53c109f7dbe617af11f8ddcda039b3671f8faed15eaa4bd8bcfb8
2022-05-18T04:37:44.7082328Z deleted: sha256:18725d5d610c69999c371599c7dfdbfca81db91bdce7335aae3dc49348367ab8
2022-05-18T04:37:44.7082642Z deleted: sha256:85f4743616e5158b5567fc794918955beb32b5236b2e10f819267c2ff313ee69
2022-05-18T04:37:44.7082943Z deleted: sha256:ab66af73e31229cc15f953e8c2279f4d766fa3c2051238adbfa58a3665de7f65
2022-05-18T04:37:44.7083270Z deleted: sha256:1caf1fe31f6355fc053d0a885626cb13472cc19222869ad8e6977bc1151830b4
2022-05-18T04:37:44.7083731Z deleted: sha256:c330d81b0faf6dede539990f626ac105eec327f65ab36c1a7a374f7242467013
2022-05-18T04:37:44.7084049Z deleted: sha256:718e0b324b58098941fadb420dd7e57dbbdd3b591289a143fbfe5ec42979f7f6
2022-05-18T04:37:44.7084367Z deleted: sha256:195247420be4cd7903121d178b00f6a257dd02af36c0129e340ac8ad968a008c
2022-05-18T04:37:44.7084674Z deleted: sha256:656c5db2781399301eb88e975b028e903a76c3fa4bdcb5ee2f601596d3770fe0
2022-05-18T04:37:44.7085003Z deleted: sha256:8aa8cf05fa63d857f6a09fc51e63e28efa3eeb017fc313091bbd2c41188ee73a
2022-05-18T04:37:44.7085327Z deleted: sha256:f3103bd0a274988a3c363a8c3e0c66cf27245bd0fbbf36f23c3a776bebce0636
2022-05-18T04:37:44.7085637Z deleted: sha256:14d2527d258aeb357a7c02ab642543354c6926e735b319b2840837e5f0bc6338
2022-05-18T04:37:44.7085955Z deleted: sha256:c1f967cb927feb91ad01be2df341c90526a0eeb449a5ba45d77955063390f5bd
2022-05-18T04:37:44.7086334Z deleted: sha256:facd48d4abdd74267bc84f37717a90597ee71abf90b442c2f080b8bff506eb28
2022-05-18T04:37:44.7086679Z deleted: sha256:36ca359a3de05a1ade3b9d47f386d0c7fdb7a2dd92ff430ac0b20252fdc971e6
2022-05-18T04:37:44.7087007Z deleted: sha256:f239740304fbce1682f9451b901322be1fd04bdba61ce3ec18219e2aa06abf92
2022-05-18T04:37:44.7087318Z deleted: sha256:95725bafda94d79e52e791051ca845246bac6420b367799457f13c9c4823e42f
2022-05-18T04:37:44.7087636Z deleted: sha256:edeffeace97c6ee0cb3c330b29296533b666ab913d0e3ce41e7b7ec02aa86a00
2022-05-18T04:37:44.7087969Z deleted: sha256:f9dc7cff308d456d8441528cfa5b049927aa044ca92450651297e1ea392b5176
2022-05-18T04:37:44.7088275Z deleted: sha256:829e5bacf960e9a03137a045e2962146d3675320f540b9f78bb656f6a5ebbd87
2022-05-18T04:37:44.7088574Z deleted: sha256:45c8bf957922e20b8cc1b780b1fa1d9cc6134f6898ac929075ebcb67f8e3d8ce
2022-05-18T04:37:44.7088898Z deleted: sha256:8e711fe9a02652cabb3d87bd91027d6f196acca0cac325f5218833bd2124edd0
2022-05-18T04:37:44.7089217Z deleted: sha256:41110f2b015c22f06f254e16ceec24e6d5f0508a3c03832e9e1bafdc9dcd6de9
2022-05-18T04:37:44.7089531Z deleted: sha256:492624ed296210ba14ce215ca0c827f0338281b43fa797db8beb8dc9eb73a075
2022-05-18T04:37:44.7089843Z deleted: sha256:1fa662cad30854c503cb8a6dca8775feb55ee9c810ab8a1964deb4df50423c59
2022-05-18T04:37:44.7090159Z deleted: sha256:0620304215b9c7b0c8a939c442cd21e16f69b3acf6e41b63db620388bcffe636
2022-05-18T04:37:44.7090489Z deleted: sha256:aad214394eed942b6cffbd34edd8d075fd9ec3cdae03021b68bb6a665df027f7
2022-05-18T04:37:44.7090789Z deleted: sha256:ef98d4551894826fa3e82876524657321978570336809b9d5f65f37ea57ba737
2022-05-18T04:37:44.7091082Z deleted: sha256:d4a7f9505f526ff752975a5b42ec0649e121dce31a1c20b64580bdb272398106
2022-05-18T04:37:44.7091393Z deleted: sha256:7c5163657fc0146a1c01bb4248aa43ef7e9f7bcdaa6476d60682c942743e7a76
2022-05-18T04:37:44.7091706Z deleted: sha256:914eca704a7c999d8554ac377319fba5d20d7bd71d037bcaea5c1789a0cd4588
2022-05-18T04:37:44.7092026Z deleted: sha256:41cc6d134c10d2b07ee7b79af7d9e2d9adaeeb66740a7f3f2b6110ff5c9b3750
2022-05-18T04:37:44.7092367Z deleted: sha256:3c52cedd16ec9a3d3ba05b9d59f95fe9e9c17b9ef45a8535780e33a4c796399f
2022-05-18T04:37:44.7092693Z deleted: sha256:b4ff4898d9d5652835190faa9f1656f944eef4b15d71d28980910a097520b29e
2022-05-18T04:37:44.7093011Z deleted: sha256:ba1a0ef9d6dacf9bd6a466603f1c0a439c5e4b16a1f08123366d98cbd451e552
2022-05-18T04:37:44.7093335Z deleted: sha256:9e86378183e56a8b8e03ac1662012715ba76ff596ac9d82493a26e2c05469e0c
2022-05-18T04:37:44.7093642Z deleted: sha256:b8f9bd44c8e3ef9185a9356770c6809d3b1e7eabe334a235bb3809c940736ee8
2022-05-18T04:37:44.7093954Z deleted: sha256:2194650d50c88f79f0c9316ff973d3fc8f05b34482c634ae5d1872d0488d6063
2022-05-18T04:37:44.7094249Z deleted: sha256:b6b59e40dec31e9e447977692ead70dc928bc273b29366a09f960ffe36615ca5
2022-05-18T04:37:44.7094564Z deleted: sha256:7538e51bd192b2c72757560bf89efb2b558e7feae722a54dce9ac5011b11334f
2022-05-18T04:37:44.7095026Z deleted: sha256:157a5446033e8978b7d89bbcd2289cd0973cee0da068e5983d6aec23abc16606
2022-05-18T04:37:44.7095465Z deleted: sha256:7886d23d8ebae5acb8cf55f4023a5f24d001deb6175bc294c49cc8426dbcd0fe
2022-05-18T04:37:44.7095924Z deleted: sha256:055c9429b696b8a0b5ae3182361238d4ad7b7ebe936dbb5329784a6e3e466eaf
2022-05-18T04:37:44.7096471Z deleted: sha256:0828f67acda3c667c57d5ee7d8c702dade92c908d6c673b041b693dd31ae1d25
2022-05-18T04:37:44.7096945Z deleted: sha256:25f905fb8bb2f1c914abedbc5059b4e47897cd0492f971d68b6977a921a35219
2022-05-18T04:37:44.7097413Z deleted: sha256:c8192f9ee988fa7475dda1364fe7643d799337ba7cbab7ef34fda310c2902122
2022-05-18T04:37:44.7097944Z deleted: sha256:d2c75ac26d00f774923ecde3f66d668f57f552b7e648bdc696922ad82dd5ae23
2022-05-18T04:37:44.7098539Z deleted: sha256:13ba83328c52b569d2601deca46a81b79edf7abd41d4b0c0b51bac7a098630df
2022-05-18T04:37:44.7099101Z deleted: sha256:05525537eae2e6755a75df8627211d016b48e97f1be4e17e41d58d7710493358
2022-05-18T04:37:44.7099626Z deleted: sha256:0214f4b057d78b44fd12702828152f67c0ce115f9346acc63acdf997cab7e7c8
2022-05-18T04:37:44.7100156Z deleted: sha256:1b9d0485372c5562fa614d5b35766f6c442539bcee9825a6e90d1158c3299a61
2022-05-18T04:37:44.7100810Z deleted: sha256:3c0f34be6eb98057c607b9080237cce0be0b86f52d51ba620dc018a3d421baea
2022-05-18T04:37:44.7101323Z deleted: sha256:be96a3f634de79f523f07c7e4e0216c28af45eb5776e7a6238a2392f71e01069
2022-05-18T04:37:44.7101585Z 
2022-05-18T04:37:44.7101724Z Total reclaimed space: 5.848GB
2022-05-18T04:37:44.7145934Z Post job cleanup.
2022-05-18T04:37:44.7173025Z Post job cleanup.
2022-05-18T04:37:44.8118434Z [command]/usr/bin/git version
2022-05-18T04:37:44.8154523Z git version 2.32.0
2022-05-18T04:37:44.8191883Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/69cb0d1a-28c0-49e6-9533-4d4a4d64ad5d' before making global git config changes
2022-05-18T04:37:44.8192298Z Adding repository directory to the temporary git global config as a safe directory
2022-05-18T04:37:44.8198299Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch
2022-05-18T04:37:44.8234046Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand
2022-05-18T04:37:44.8262989Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :
2022-05-18T04:37:44.8519030Z Entering 'android/libs/fbjni'
2022-05-18T04:37:44.8553411Z Entering 'third_party/FP16'
2022-05-18T04:37:44.8589047Z Entering 'third_party/FXdiv'
2022-05-18T04:37:44.8620482Z Entering 'third_party/NNPACK'
2022-05-18T04:37:44.8654148Z Entering 'third_party/QNNPACK'
2022-05-18T04:37:44.8687251Z Entering 'third_party/XNNPACK'
2022-05-18T04:37:44.8732680Z Entering 'third_party/benchmark'
2022-05-18T04:37:44.8765526Z Entering 'third_party/cpuinfo'
2022-05-18T04:37:44.8798595Z Entering 'third_party/cub'
2022-05-18T04:37:44.8832232Z Entering 'third_party/cudnn_frontend'
2022-05-18T04:37:44.8869741Z Entering 'third_party/eigen'
2022-05-18T04:37:44.8904172Z Entering 'third_party/fbgemm'
2022-05-18T04:37:44.8935829Z Entering 'third_party/fbgemm/third_party/asmjit'
2022-05-18T04:37:44.8968027Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2022-05-18T04:37:44.9001212Z Entering 'third_party/fbgemm/third_party/googletest'
2022-05-18T04:37:44.9034859Z Entering 'third_party/flatbuffers'
2022-05-18T04:37:44.9069335Z Entering 'third_party/fmt'
2022-05-18T04:37:44.9101830Z Entering 'third_party/foxi'
2022-05-18T04:37:44.9133392Z Entering 'third_party/gemmlowp/gemmlowp'
2022-05-18T04:37:44.9165856Z Entering 'third_party/gloo'
2022-05-18T04:37:44.9198582Z Entering 'third_party/googletest'
2022-05-18T04:37:44.9231788Z Entering 'third_party/ideep'
2022-05-18T04:37:44.9264106Z Entering 'third_party/ideep/mkl-dnn'
2022-05-18T04:37:44.9297188Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN'
2022-05-18T04:37:44.9334979Z Entering 'third_party/ios-cmake'
2022-05-18T04:37:44.9366976Z Entering 'third_party/kineto'
2022-05-18T04:37:44.9399152Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2022-05-18T04:37:44.9432362Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2022-05-18T04:37:44.9465810Z Entering 'third_party/nccl/nccl'
2022-05-18T04:37:44.9498408Z Entering 'third_party/neon2sse'
2022-05-18T04:37:44.9530369Z Entering 'third_party/onnx'
2022-05-18T04:37:44.9574003Z Entering 'third_party/onnx/third_party/benchmark'
2022-05-18T04:37:44.9606794Z Entering 'third_party/onnx/third_party/pybind11'
2022-05-18T04:37:44.9640592Z Entering 'third_party/onnx-tensorrt'
2022-05-18T04:37:44.9673866Z Entering 'third_party/onnx-tensorrt/third_party/onnx'
2022-05-18T04:37:44.9710146Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2022-05-18T04:37:44.9741891Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2022-05-18T04:37:44.9774012Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2022-05-18T04:37:44.9809907Z Entering 'third_party/pocketfft'
2022-05-18T04:37:44.9842806Z Entering 'third_party/protobuf'
2022-05-18T04:37:44.9878887Z Entering 'third_party/protobuf/third_party/benchmark'
2022-05-18T04:37:44.9911775Z Entering 'third_party/protobuf/third_party/googletest'
2022-05-18T04:37:44.9946139Z Entering 'third_party/psimd'
2022-05-18T04:37:44.9978205Z Entering 'third_party/pthreadpool'
2022-05-18T04:37:45.0010276Z Entering 'third_party/pybind11'
2022-05-18T04:37:45.0043672Z Entering 'third_party/python-enum'
2022-05-18T04:37:45.0075525Z Entering 'third_party/python-peachpy'
2022-05-18T04:37:45.0107420Z Entering 'third_party/python-six'
2022-05-18T04:37:45.0139743Z Entering 'third_party/sleef'
2022-05-18T04:37:45.0172859Z Entering 'third_party/tbb'
2022-05-18T04:37:45.0206683Z Entering 'third_party/tensorpipe'
2022-05-18T04:37:45.0239684Z Entering 'third_party/tensorpipe/third_party/googletest'
2022-05-18T04:37:45.0271512Z Entering 'third_party/tensorpipe/third_party/libnop'
2022-05-18T04:37:45.0303493Z Entering 'third_party/tensorpipe/third_party/libuv'
2022-05-18T04:37:45.0336084Z Entering 'third_party/tensorpipe/third_party/pybind11'
2022-05-18T04:37:45.0367927Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2022-05-18T04:37:45.0402467Z Entering 'third_party/zstd'
2022-05-18T04:37:45.0452047Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader
2022-05-18T04:37:45.0476273Z http.https://github.com/.extraheader
2022-05-18T04:37:45.0484039Z [command]/usr/bin/git config --local --unset-all http.https://github.com/.extraheader
2022-05-18T04:37:45.0517001Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :
2022-05-18T04:37:45.0766830Z Entering 'android/libs/fbjni'
2022-05-18T04:37:45.0786231Z http.https://github.com/.extraheader
2022-05-18T04:37:45.0812058Z Entering 'third_party/FP16'
2022-05-18T04:37:45.0832412Z http.https://github.com/.extraheader
2022-05-18T04:37:45.0858141Z Entering 'third_party/FXdiv'
2022-05-18T04:37:45.0878936Z http.https://github.com/.extraheader
2022-05-18T04:37:45.0904568Z Entering 'third_party/NNPACK'
2022-05-18T04:37:45.0923795Z http.https://github.com/.extraheader
2022-05-18T04:37:45.0950570Z Entering 'third_party/QNNPACK'
2022-05-18T04:37:45.0969843Z http.https://github.com/.extraheader
2022-05-18T04:37:45.0995509Z Entering 'third_party/XNNPACK'
2022-05-18T04:37:45.1014626Z http.https://github.com/.extraheader
2022-05-18T04:37:45.1049610Z Entering 'third_party/benchmark'
2022-05-18T04:37:45.1070079Z http.https://github.com/.extraheader
2022-05-18T04:37:45.1096344Z Entering 'third_party/cpuinfo'
2022-05-18T04:37:45.1118003Z http.https://github.com/.extraheader
2022-05-18T04:37:45.1143119Z Entering 'third_party/cub'
2022-05-18T04:37:45.1183072Z http.https://github.com/.extraheader
2022-05-18T04:37:45.1207897Z Entering 'third_party/cudnn_frontend'
2022-05-18T04:37:45.1227702Z http.https://github.com/.extraheader
2022-05-18T04:37:45.1257599Z Entering 'third_party/eigen'
2022-05-18T04:37:45.1277691Z http.https://github.com/.extraheader
2022-05-18T04:37:45.1305501Z Entering 'third_party/fbgemm'
2022-05-18T04:37:45.1324600Z http.https://github.com/.extraheader
2022-05-18T04:37:45.1351195Z Entering 'third_party/fbgemm/third_party/asmjit'
2022-05-18T04:37:45.1369324Z http.https://github.com/.extraheader
2022-05-18T04:37:45.1394791Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2022-05-18T04:37:45.1413878Z http.https://github.com/.extraheader
2022-05-18T04:37:45.1439782Z Entering 'third_party/fbgemm/third_party/googletest'
2022-05-18T04:37:45.1458775Z http.https://github.com/.extraheader
2022-05-18T04:37:45.1486258Z Entering 'third_party/flatbuffers'
2022-05-18T04:37:45.1506432Z http.https://github.com/.extraheader
2022-05-18T04:37:45.1533952Z Entering 'third_party/fmt'
2022-05-18T04:37:45.1553784Z http.https://github.com/.extraheader
2022-05-18T04:37:45.1578552Z Entering 'third_party/foxi'
2022-05-18T04:37:45.1598150Z http.https://github.com/.extraheader
2022-05-18T04:37:45.1624069Z Entering 'third_party/gemmlowp/gemmlowp'
2022-05-18T04:37:45.1643514Z http.https://github.com/.extraheader
2022-05-18T04:37:45.1669298Z Entering 'third_party/gloo'
2022-05-18T04:37:45.1688744Z http.https://github.com/.extraheader
2022-05-18T04:37:45.1714761Z Entering 'third_party/googletest'
2022-05-18T04:37:45.1733388Z http.https://github.com/.extraheader
2022-05-18T04:37:45.1759036Z Entering 'third_party/ideep'
2022-05-18T04:37:45.1779301Z http.https://github.com/.extraheader
2022-05-18T04:37:45.1805141Z Entering 'third_party/ideep/mkl-dnn'
2022-05-18T04:37:45.1823880Z http.https://github.com/.extraheader
2022-05-18T04:37:45.1850430Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN'
2022-05-18T04:37:45.1870870Z http.https://github.com/.extraheader
2022-05-18T04:37:45.1902367Z Entering 'third_party/ios-cmake'
2022-05-18T04:37:45.1922695Z http.https://github.com/.extraheader
2022-05-18T04:37:45.1947539Z Entering 'third_party/kineto'
2022-05-18T04:37:45.1966685Z http.https://github.com/.extraheader
2022-05-18T04:37:45.1993214Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2022-05-18T04:37:45.2012052Z http.https://github.com/.extraheader
2022-05-18T04:37:45.2037839Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2022-05-18T04:37:45.2057491Z http.https://github.com/.extraheader
2022-05-18T04:37:45.2084934Z Entering 'third_party/nccl/nccl'
2022-05-18T04:37:45.2104655Z http.https://github.com/.extraheader
2022-05-18T04:37:45.2130151Z Entering 'third_party/neon2sse'
2022-05-18T04:37:45.2149887Z http.https://github.com/.extraheader
2022-05-18T04:37:45.2175387Z Entering 'third_party/onnx'
2022-05-18T04:37:45.2194896Z http.https://github.com/.extraheader
2022-05-18T04:37:45.2232032Z Entering 'third_party/onnx/third_party/benchmark'
2022-05-18T04:37:45.2251411Z http.https://github.com/.extraheader
2022-05-18T04:37:45.2276994Z Entering 'third_party/onnx/third_party/pybind11'
2022-05-18T04:37:45.2296617Z http.https://github.com/.extraheader
2022-05-18T04:37:45.2324507Z Entering 'third_party/onnx-tensorrt'
2022-05-18T04:37:45.2344287Z http.https://github.com/.extraheader
2022-05-18T04:37:45.2369838Z Entering 'third_party/onnx-tensorrt/third_party/onnx'
2022-05-18T04:37:45.2389449Z http.https://github.com/.extraheader
2022-05-18T04:37:45.2419833Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2022-05-18T04:37:45.2440474Z http.https://github.com/.extraheader
2022-05-18T04:37:45.2466009Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2022-05-18T04:37:45.2485558Z http.https://github.com/.extraheader
2022-05-18T04:37:45.2511329Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2022-05-18T04:37:45.2530678Z http.https://github.com/.extraheader
2022-05-18T04:37:45.2560664Z Entering 'third_party/pocketfft'
2022-05-18T04:37:45.2580039Z http.https://github.com/.extraheader
2022-05-18T04:37:45.2606021Z Entering 'third_party/protobuf'
2022-05-18T04:37:45.2625692Z http.https://github.com/.extraheader
2022-05-18T04:37:45.2654240Z Entering 'third_party/protobuf/third_party/benchmark'
2022-05-18T04:37:45.2673969Z http.https://github.com/.extraheader
2022-05-18T04:37:45.2698882Z Entering 'third_party/protobuf/third_party/googletest'
2022-05-18T04:37:45.2719595Z http.https://github.com/.extraheader
2022-05-18T04:37:45.2746640Z Entering 'third_party/psimd'
2022-05-18T04:37:45.2766927Z http.https://github.com/.extraheader
2022-05-18T04:37:45.2792452Z Entering 'third_party/pthreadpool'
2022-05-18T04:37:45.2812360Z http.https://github.com/.extraheader
2022-05-18T04:37:45.2837522Z Entering 'third_party/pybind11'
2022-05-18T04:37:45.2857078Z http.https://github.com/.extraheader
2022-05-18T04:37:45.2883774Z Entering 'third_party/python-enum'
2022-05-18T04:37:45.2903451Z http.https://github.com/.extraheader
2022-05-18T04:37:45.2928942Z Entering 'third_party/python-peachpy'
2022-05-18T04:37:45.2948939Z http.https://github.com/.extraheader
2022-05-18T04:37:45.2974258Z Entering 'third_party/python-six'
2022-05-18T04:37:45.2993366Z http.https://github.com/.extraheader
2022-05-18T04:37:45.3018819Z Entering 'third_party/sleef'
2022-05-18T04:37:45.3037528Z http.https://github.com/.extraheader
2022-05-18T04:37:45.3063030Z Entering 'third_party/tbb'
2022-05-18T04:37:45.3082901Z http.https://github.com/.extraheader
2022-05-18T04:37:45.3109930Z Entering 'third_party/tensorpipe'
2022-05-18T04:37:45.3129612Z http.https://github.com/.extraheader
2022-05-18T04:37:45.3155757Z Entering 'third_party/tensorpipe/third_party/googletest'
2022-05-18T04:37:45.3174695Z http.https://github.com/.extraheader
2022-05-18T04:37:45.3200210Z Entering 'third_party/tensorpipe/third_party/libnop'
2022-05-18T04:37:45.3219389Z http.https://github.com/.extraheader
2022-05-18T04:37:45.3245154Z Entering 'third_party/tensorpipe/third_party/libuv'
2022-05-18T04:37:45.3264795Z http.https://github.com/.extraheader
2022-05-18T04:37:45.3290577Z Entering 'third_party/tensorpipe/third_party/pybind11'
2022-05-18T04:37:45.3311222Z http.https://github.com/.extraheader
2022-05-18T04:37:45.3336685Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2022-05-18T04:37:45.3355981Z http.https://github.com/.extraheader
2022-05-18T04:37:45.3385044Z Entering 'third_party/zstd'
2022-05-18T04:37:45.3403698Z http.https://github.com/.extraheader
2022-05-18T04:37:45.3623225Z Cleaning up orphan processes